- MICROCOPY  RESOLUTION  TEST  CHART 


NATIONAL  BURLAU  01  STANDARDS  19n<  A 


ODC  FILE  COPY  AD  AO  471  27 


NAVAL  POSTGRADUATE  SCHOOL 

Monterey,  California 


THE  MEASUREMENT  OF  EFFORTS  TO  ENHANCE  THE 

EFFECTIVENESS  OF  NAVFAC  RDT&E  INVESTMENTS 

by 

Eugene  H.  Early 

June  1977 

Thesis  Advisors:  J.  W.  Creighton 

J.  A.  Jolly 

Approved  for  public  release;  distribution  unlimited. 


SECURITY  CLASSIFICATION  of  THIS  FAOS  r*h«  Data  Bn lara« 


REPORT  DOCUMENTATION  PAGE 


READ  INSTRUCTIONS 
BEFORE  COMPLETING  FORM 


IRlENT'l  catalog  number 


A-  The  Measurement  of  Efforts  to  Enhar^pe  \ /[Master 1 s yfhesis 
the  Effectiveness  of  NAVFAC  RDTMe  ^ 

\ Investments , 


FORMING  ORG.  RENO* 


7.  AuTHORfa; 


fO) 


h./e 


Eugene  H./ Early 


».  NERFORMING  ORGANIZATION  N AM  E ANO  AOORESS 

Naval  Postgraduate  School  ■ 
Monterey,  California  93940 


II.  CONTROLLING  OFFICE  NAME  ANO  AOORESS 

Naval  Postgraduate  School 
Monterey,  California  93940 


NCY  NAME  A AOORESXff  rfffferwif  trmm  Controlling  Ollles) 


OECL  ASSI  Pi  CATION/  DOWNGRADING 
SCHEDULE 


l«.  01  ST Rl GU Tl ON  STATEMENT  (oi  this  R sport) 


Approved  for  public  release;  distribution  unlimited. 


<7.  OIST  Ri  BUT  ION  STATEMENT  (ml  (Aa  aAatraal  aniaraA  la  Utmmk  30,  II  rnitlmmmnt  Irmm  *m*mn) 


I*.  KEY  BOROS  (Cmnllmtm  an  rararaa 

Quantification 

Research 

Benefit 

Sampling 

Estimates 


20.  ABSTRACT  (Csntlmf  on  room—  i 


• II  nocmmmmry  Usmtlff  by  blmsk  numb—) 


Utilization 

Transfer 

Information 

Technology 

Feedback 


• II  i—cmssmy  l Pont  l tr  by  blook  i 


Transfer  of  Technology 
Cost  Effectiveness 
Information  Systems 
Computer 
Data  Bases 


— ^ The  concept  of  using  sampling  procedures  and  user  evaluations 
to  measure  the  effectiveness  of  a small  service  program  was  intro 
duced.  Evaluations  were  based  on  the  benefit  of  information 
transferred  to  end  users.  The  feasibility  of  this  approach  was 
demonstrated  using  the  evaluated  benefit  of  randomly  selected 
service  requests  as  a basis  for  estimating  the  total  benefit  and 


00  i^STn  U73  COITION  OF  I NOV  I*  OBSOLETE 

(Page  1)  */"  0 10  3*  0 14*  SS0  I | l 


UNCLASSIFIED 

SECURITY  CLASSIFICATION  OF  TmII  FAGS  r**»"  £>•** 


J L >4 SXb 


UNCLASSIFIED 


^return  on  investment  (ROI)  for  the  Assistance  Program  during  the 
fourth  quarter  of  FY-1976. 

The  concept  of  using  subjective  evaluations  to  measure  effec- 
tiveness in  a more  timely  manner  was  also  introduced  and  demon- 
strated to  be  a useful  approach  to  estimate  ROI  with  reasonable 
accuracy. 

An  automated  information  system  using  the  Statistical  Package 
for  the  Social  Sciences  was  also  expanded  in  an  operational  envi- 
ronment to  accommodate  the  data  processing  requirements  of  an 
effectiveness  measuring  system  for  the  Facilities  Engineering 
Support  Office,  a service  organization  of  the  Navy's  Civil 
Engineering  Laboratory. 


2 


Approved  for  public  release;  distribution  unlimited 


The  Measurement  of  Efforts  to  Enhance  the  Effectiveness 
of  NAVFAC  RDT&E  Investments 


by 

Eugene  H.  Early 
Civil  Engineering  Laboratory 
Naval  Construction  Battalion  Center 
Port  Hueneme,  California 
B.S.C.E.,  University  of  California,  1954 


Submitted  in  partial  fulfillment 
of  the  requirements  for  the  degree  of 


MASTER  OF  SCIENCE  IN  MANAGEMENT 

from  the 

NAVAL  POSTGRADUATE  SCHOOL 
June  1977 


ABSTRACT 


The  concept  of  using  sampling  procedures  and  user  evalua- 
tions to  measure  the  effectiveness  of  a small  service  program 
was  introduced.  Evaluations  were  based  on  the  benefit  of 

j 

information  transferred  to  end  users.  The  feasibility  of  this 
approach  was  demonstrated  using  the  evaluated  benefit  or  ran- 
domly selected  service  requests  as  a basis  for  estimating  the 
total  benefit  and  return  on  investment  (ROI)  for  the  Assist- 
ance Program  during  the  fourth  quarter  of  FY-1976. 

The  concept  of  using  subjective  evaluations  to  measure 
effectiveness  in  a more  timely  manner  was  also  introduced  and 
demonstrated  to  be  a useful  approach  to  estimate  ROI  with  rea- 
sonable accuracy. 

An  automated  information  system  using  the  Statistical 
Package  for  the  Social  Sciences  was  also  expanded  in  an  opera- 
tional environment  to  accommodate  the  data  processing  require- 
ments of  an  effectiveness  measuring  system  for  the  Facilities 
Engineering  Support  Office,  a service  organization  of  the 
Navy's  Civil  Engineering  Laboratory. 
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A.  PERFORMANCE  MEASURES 

The  need  to  measure  the  performance  of  programmed  efforts 
has  always  been  of  concern  to  the  manager,  whether  he  is  lo- 
cated at  the  operational  or  headquarters  level.  In  these 
days  of  limited  resources,  the  need  has  become  more  critical. 
Two  common  measures  of  performance  are  efficiency  and  effec- 
tiveness. Drucker  [Ref.  1],  in  his  discussion  of  the  differ- 
ence between  efficiency  and  effectiveness,  stated: 

"Effectiveness  is  the  function  of  success — efficiency  is 
a minimum  condition  for  survival  after  success  has  been 
achieved.  Efficiency  is  concerned  with  doing  things 
right.  Effectiveness  is  doing  the  right  things. 

Efficiency  concerns  itself  with  the  input  of  effort  into 
all  areas  of  activity.  Effectiveness,  however,  starts 
out  with  the  realization  that  in  business,  as  in  any  other 
social  organism,  10  or  15  percent  of  the  phenomena — such 
as  products,  orders,  customers,  markets,  or  people — 
produce  80  to  90  percent  of  the  results.  The  other  85 
to  90  percent  of  the  phenomena,  no  matter  how  efficiently 
taken  care  of,  produce  nothing  but  costs  (which  are  always 
proportionate  to  transactions,  that  is  busy-ness)." 

This  study  concentrates  on  efforts  to  measure  effective- 
ness. However,  the  effectiveness  of  performance  in  certain 
areas  of  operations  has  essentially  eluded  evaluation  efforts. 
This  is  generally  true  in  the  area  of  research  and 
development  [Ref.  2]. 

B.  MEASURING  EFFECTIVENESS 

In  order  to  evaluate  the  effectiveness  of  research  efforts, 
the  manager  must  first  decide  what  is  going  to  be  evaluated. 

In  the  case  of  a particular  program,  the  purpose  of  that 
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is  valuable.  The  manager  must  then  either  estimate  the 
potential  value  or  benefit  of  future  events,  assuming  that 
those  events  will  in  fact  take  place,  or  wait  until  the 
results  of  the  research  are  known.  At  some  later  date,  if 
the  results  are  committed  to  practice,  he  will  assess  the 
benefit  that  has  been  realized.  The  former  approach  is 
frought  with  subjectivity.  The  later  approach  entails  an 
unacceptable  waiting  period,  as  well  as  a significant  degree 
of  subjectivity.  Neither  approach  is  desirable. 

Coupled  with  the  inherent  problems  of  subjectivity  and 
timeliness,  is  the  question  of  which  evaluation  techniques 
to  use  to  measure  effectiveness.  In  the  area  of  research 
and  development,  they  do  not  exist,  for  the  most  part, 
although  progress  has  been  made  in  certain  limited  areas. 
Finally,  the  measuring  system  must  not  require  an  inordin- 
ately large  level  of  effort  compared  to  the  effort  being 
evaluated.  A small  organization  or  a small  program  should, 
therefore,  only  allocate  a limited  amount  of  resources  to 
measuring  effectiveness.  Managers  faced  with  these  obstacles 
and  constraints  are  often  inclined  to  abandon  hope  of  quanti- 
tatively measuring  program  effectiveness. 

An  alternative  to  abandoning  hope,  is  to  consider 
adapting  those  techniques  for  evaluating  benefit  which  are 
workable,  and  through  the  use  of  a sampling  approach,  obtain 
sufficient  data  to  estimate  total  benefit  at  a level  of 
accuracy  commensurate  with  the  resources  available  for 
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measuring  effectiveness.  The  success  of  such  an  effort 
offers  the  manager  a capability  that  does  not  currently 
exist,  at  an  affordable  price. 

C.  PURPOSE  OF  THE  STUDY 

The  objective  of  this  study  is  to  develop  an  effective- 
ness measuring  system  based  on  sampling  procedures  and  to 
demonstrate  that  it  is  a feasible  approach  to  meeting  manage- 
ment requirements  for  information  on  the  effectiveness  of  a 
small  service  program.  Such  a system  will  provide  a viable 
alternative  in  those  situations  where  the  lack  of  sufficient 
resources,  or  the  relative  magnitude  of  the  program  effort, 
precludes  the  accomplishment  of  full-scale  evaluation  efforts. 
Specifically,  a previously  developed  evaluation  technique 
will  be  optimized  and  applied  to  a selected  number  of  indivi- 
dual service  requests  to  provide  the  basis  for  estimating 
the  annual  return  on  investment  (ROI)  for  the  whole  service 
program.  It  will  be  demonstrated  that  the  data  resulting 
from  the  evaluation  of  selected  individual  asssistance 
requests,  when  processed  by  a computer-aided  management 
information  system,  provides  a reasonably  accurate  and  econ- 
omically affordable  basis  for  estimating  the  annual  ROI  of 
resources  expended  by  the  Navy's  Civil  Engineering  Laboratory 
in  executing  the  Naval  Facilities  Engineering  Command's 
Assistance  Program. 
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II.  BACKGROUND 


A.  NAVAL  FACILITIES  ENGINEERING  COMMAND 

The  Naval  Facilities  Engineering  Command  ( NAVFAC ) provides 
engineering,  material  and  equipment  support  to  the  Chief  of 
Naval  Operations,  the  Operating  Forces  of  the  Navy,  the 
Marine  Corps,  components  of  the  Naval  Material  Command,  and 
other  offices  [Ref.  3].  NAVFAC's  Research  Program  is  directed 
primarily  towards  items  of  new  or  improved  materials,  equip- 
ment or  engineering  techniques  which  will  significantly  im- 
prove solutions  to  specific  engineering  problems  pertaining 
to  the  technical  planning,  design,  construction,  operation 
and  maintenance  of  the  shore  activities  and  fixed  surface 
and  subsurface  structures  of  the  Navy  [Ref.  3]. 

NAVFAC's  Research  Program  is  specifically  administered 
by  the  Assistant  Commander  for  Research  and  Development 
(Code  03) . NAVFAC  Code  03  therefore  has  the  responsibility 
to  ensure  that  the  output  of  Research  and  Development  ( R&D) 
is  transferred  to  shore  activities  of  the  Navy  and  to  ensure 
that  maximum  effectiveness  is  achieved  from  R&D  investments. 

NAVFAC's  link  to  the  shore  activities  is  primarily 
through  its  Engineering  Field  Divisions  (EFDs) , its  Public 
Works  Centers  (PWCs) , its  construction  program  administered 
by  Officers  in  Charge  of  Construction  (OICCs)  and  Resident 
Officers  in  Charge  of  Construction  (ROICCs) , and  Public 
Works  Departments  of  individual  shore  activities.  Figure  1 
shows  these  organizational  and  technical  relationships. 
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A major  portion  of  NAVFAC's  research  program  is  assigned 
to  and  executed  by  the  Navy's  Civil  Engineering  Laboratory 
(CEL) . 

B.  CIVIL  ENGINEERING  LABORATORY 

The  Civil  Engineering  Laboratory  (CEL)  is  under  the 
administrative  control  of  the  Naval  Construction  Battalion 
Center,  Port  Hueneme,  California,  and  is  the  principal  RDT&E 
Center  for  shore  and  fixed  surface  and  subsurface  ocean 
facilities  and  for  the  Navy  and  Marine  Corps  Construction 
Forces  [Ref.  4].  The  staff  of  CEL  consists  of  approximately 
300  personnel,  175  of  which  are  professional  engineers  and 
scientists,  representing  a large  variety  of  disciplines. 

Funding  for  CEL's  FY-76  RDT&E  program  amounted  to  13 
million  dollars  with  approximately  70  percent  of  the  program 
supported  by  Navy  RDT&E  funds.  The  program  represented 
approximately  250  individual  efforts,  called  work  units.  One 
of  these  work  units  was  RDT&E  Assistance  to  Naval  Shore 
Activities . 

C.  ASSISTANCE  PROGRAM 

The  work  unit  RDT&E  Assistance  to  Naval  Shore  Activities, 
referred  to  as  the  Assistance  Program  in  the  remainder  of 
this  report,  was  initiated  at  CEL  in  1963  using  NAVFAC 
Exploratory  Development  (6.2  category  RDT&E)  funds.  The  pur- 
pose of  the  Assistance  Program  is  to  provide  quick-response 
service,  upon  request  and  at  no  cost  to  the  requestor,  dir- 
ectly to  users  located  at  NAVFAC  field  activities  and  Public 
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Works  Departments  of  shore  activities  throughout  the  Naval 
Shoi-1-  Establishment.  These  efforts  normally  take  the  form 
of  short-term  investigations,  analyses  and  tests  to  determine 
the  relative  value  and  suitability  of  new  materials,  equipment, 
processes,  and  construction  and  maintenance  procedures  [Ref. 

5].  The  key  words  in  this  statement  are  "to  provide  services 
upon  request."  Thus  the  Assistance  Program  is  driven  by  the 
expressed  needs  of  its  users. 

In  the  process  of  assisting  shore  activities,  CEL 
researchers  provide  field  personnel  with  an  early  awareness 
of  current  R&D  efforts  and  preliminary  and  final  results  that 
relate  to  their  areas  of  expressed  interest.  This  results  in 
narrowing  the  gap  between  research  and  use  by  putting  the 
results  of  research  to  work  at  the  earliest  practicable  time 
to  solve  current  problems  being  experienced  by  field  personnel 
involved  in  facilities  engineering  matters. 

During  FY-76,  a review  of  Navy  RDT&E  programs  led  to  a 
decision  by  the  Director  of  Defense  Research  and  Engineering 
that  the  use  of  Navy  Exploratory  Development  funds  was  not 
appropriate  for  the  provision  of  assistance  to  Naval  shore 
activities.  Accordingly,  an  alternative  funding  source  had 
to  be  developed  if  the  Assistance  Program  was  to  be  continued 
after  FY-76. 

After  reviewing  the  situation,  NAVFAC  management  decided 
that  there  was  sufficient  benefit  in  providing  assistance  to 
Naval  shore  activities  and  that  these  efforts  should  be  con- 
tinued using  mission  management  funds.  Although  performance 
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measures  applied  to  CEL  assistance  efforts  related  to  activity 
and  growth  in  usage,  rather  than  effectiveness,  this  informa- 
tion was  used  as  input  to  this  NAVFAC  decision.  These  per- 
formance measures  are  discussed  in  some  detail  in  Appendix  A. 

D.  FACILITIES  ENGINEERING  SUPPORT  OFFICE 

The  Facilities  Engineering  Support  Office  (FESO)  was 
established  by  CEL  and  funded  by  NAVFAC  to  perform  the  func- 
tion of  coordinating  services  and  communications  related  to 
providing  assistance  to  shore  activities.  This  was  a reali- 
zation of  management  desire  to  focus  attention  on  this  aspect 
of  CEL's  efforts  and  to  ensure  that  timely  and  effective 
laboratory  support  was  provided  in  response  to  field  needs. 

The  function  was  placed  in  CEL's  organization  as  a staff  com- 
ponent of  the  Technical  Director.  This  location  reflects 
management  interest  and  dedication  to  this  service  and  allows 
for  easy  and  direct  access  to  CEL's  primary  resource — its 
people.  The  basic  organization  of  CEL  is  shown  in  Figure  2. 
The  primary  FESO  functions  related  to  the  Assistance  Program 
are  outlined  in  Figure  3. 

NAVFAC 's  decision  to  continue  the  Assistance  Program 
utilizing  mission  management  funds  was  based  on  performance 
measures  of  program  activity  mentioned  earlier  and  other  sub- 
jective judgements  of  the  value  of  the  program.  Accordingly, 
the  need  to  quantify  the  effectiveness  of  the  Assistance  Pro- 
gram became  a matter  of  increased  management  interest  and 
importance.  Developing  a system  to  measure  the  effectiveness 
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ASSISTANT  Of  MCI  A IN  CHAASE 


Figure  2 Organization  of  the  Civil  Engineering  Laboratory 


The  functions  of  the  Facilities  Engineering  Support 
Office  (FESO)  related  to  providing  RDT&E  Assistance  to  Naval 
Shore  Activities  are: 

1.  Ensure  that  timely  engineering  support  services  are 
provided  to  NAVFAC  and  Public  Works  field  activities 
throughout  the  Shore  Establishment  on  facility  engineering 
matters . 

2.  Serve  as  a point  of  contact  at  CEL  for  liaison  with 
field  activities  on  facility  engineering  RDT&E  matters. 

3.  Search  out,  develop,  and  maintain  a knowledge  of  the 
assistance  needs  of  these  field  activities  and  provide 
them  with  a knowledge  of  CEL's  capabilities  and  interest 
in  meeting  their  needs. 

4.  Act  as  a project  coordinator  on  assistance  projects 
for  field  activities  to  ensure  timely  and  effective  labor- 
atory responsiveness  to  these  needs. 

5.  Act  as  a focal  point  for  correspondence  relating  to 
assistance  to  field  activities,  including  promotion  of 
direct  contact  between  field  personnel  and  individual  mem- 
bers of  CEL's  technical  staff. 

6.  Effect  utilization  of  currently  available  research 
results  as  a means  of  meeting  the  needs  of  field  activities. 

7.  Formulate  plans  for  RDT&E  support  of  field  units  by 
CEL  where  a solution  is  not  currently  available,  including 
fostering  the  initiating  of  RDT&E  task  proposals  and  ob- 
taining field  activity  sponsorship  of  such  RDT&E  proposals. 

8.  Study  the  spectra  of  assistance  needs  of  the  field 
activities,  synthesize  these  needs  and  their  solutions, 

and  disseminate  findings  to  these  field  activities  [Ref.  4]. 


Figure  3.  FESO  Functions  Related  to  The  Assistance  Program 
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III.  SCOPE  OF  THE  WORK 


A.  ASSUMPTIONS 

The  basic  assumption  of  this  study  is  that  a pressing 
need  exists  to  obtain  quantified  information  on  the  effective- 
ness of  the  Assistance  Program  on  an  on-going  basis. 

A parallel  assumption  is  that  the  expense  of  evaluating 
all  assistance  requests,  on  an  on-going  basis,  is  not  war- 
ranted by  the  level  of  effort  expended  in  providing  assistance 
services.  Even  if  it  were  justified,  the  effort  required  for 
such  an  evaluation  would  exceed  the  resources  available  to 
the  FESO. 

The  prevailing  rationale  is  that  the  primary  purpose  of 
the  Assistance  Program  is  to  provide  service  to  field  users. 
Secondarily,  the  Assistance  Program  should  be  capable  of  pro- 
ducing an  acceptable  level  of  benefit  for  those  users,  which 
at  a minimum  would  require  that  it  pay  for  itself,  or  be  self- 
supporting.  Of  course,  this  must  be  taken  figuratively, 
since  there  is  no  exchange  of  funds  between  shore  activities 
and  CEL  with  regard  to  the  Assistance  Program.  In  the  case 
of  commercial  businesses,  the  final  criteria  for  benefit, 
stated  in  terms  of  profit  and  loss,  is  return  on  investment. 
This  criteria  could  be  approximated  for  the  Assistance  Pro- 
gram by  a measure  of  discounted  present  value  of  estimates 
of  benefit  provided  by  the  users  of  the  service.  Accordingly, 
the  Assistance  Program  should  produce  a minimum  ROI  of  1.0. 
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The  need  to  generate  and  process  evaluation  data  with  a 
relatively  modest  effort  requires  adoption  of  a computer- 
based  effectiveness  measuring  system.  However,  the  system 
must  be  operable  with  the  data  processing  capability  that  is 
currently  available  to  CEL.  The  system  should  be  based  on 
evaluating  samples  of  the  total  requests  received.  The  sys- 
tem should  be  flexible  to  allow  evaluation  of  a variable 
number  of  assistance  requests,  or  all  requests  for  that 
matter . 

There  are  certain  risks  associated  with  the  gathering  of 
information  related  to  the  value  of  a particular  effort.  In 
the  case  of  the  Assistance  Program  these  risks  could  material- 
ize in  the  form  of  negative  attitudes  toward  CEL,  toward  the 
Assistance  Program,  or  toward  participating  individuals.  The 
creation  of  such  negative  attitudes  would  have  a debilitating 
effect  on  the  Assistance  Program  resulting  in  the  decreased 
use  of  CEL  as  a resource  by  potential  users,  and  decreased 
interest  in  participating,  or  in  recording  participation,  by 
CEL  personnel.  Therefore,  only  those  assistance  efforts 
which  are  judged  to  provide  measurable  benefit  should  be  in- 
cluded in  field  evaluations  of  benefit. 

B.  LIMITATIONS 

The  measuring  system  estimates  the  effectiveness  of  the 
Assistance  Program  during  a particular  period  of  time.  It 
does  not  measure  actual  benefit.  This  would  require  consider- 
able effort  over  extended  periods  of  time.  It  does  measure 
a collective  expression  of  perceived  benefit  that  users  are 
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willing  to  make,  stated  in  terms  of  estimated  return  on 
investment. 

Evaluating  the  benefits  of  a sample  of  the  total  requests 
received  during  a given  period  of  time  will  not  provide  the 
basis  for  an  exact  value  for  the  estimated  total  benefit 
during  that  period.  The  benefits  of  the  individual  assist- 
ance requests  are  normally  distributed  continuous  variables. 

At  best,  these  benefit  values  can  be  used  to  develop  a reli- 
able estimator  of  the  estimated  total  benefits  that  would 
accumulate  during  that  period,  with  a known  level  of  accuracy. 
For  example,  a significantly  large  benefit,  compared  to  other 
benefits,  might  not  be  included  in  the  sample  and,  therefore, 
would  not  influence  the  calculation  of  ROI . This  would  result 
in  understating  the  range  of  estimated  ROI. 

There  is  a risk  in  connection  with  the  possible  misuse 
of  ROI  data.  It  should  be  clearly  understood  that  the  esti- 
mated ROI  for  one  period  of  time  cannot  be  used  to  answer 
questions  about  the  level  of  benefit  achieved  by  the  Assist- 
ance Program  during  other  periods  of  time. 

There  is  a greater  risk  that  ROI  information  developed  to 
measure  the  effectiveness  of  the  Assistance  Program  during 
various  periods  of  time  might  be  used  in  such  a way  as  to  be 
inconsistent  with  the  basic  purpose  of  the  program.  Requests 
are  received  in  random  fashion,  at  the  discretion  of  request- 
ing personnel  at  shore  activities.  Therefore,  there  is  no 
logical  basis  for  assuming  that  there  should  be  a significant 
correlation  between  ROI  estimated  for  different  periods  of 
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time,  or  more  importantly,  that  the  ROI  should  continue  to 
increase  from  year  to  year.  The  later  assumption  amounts  to 
establishing  a goal  "to  make  an  increasing  larger  profit, 
while  providing  a free  service."  If  the  primary  purpose  of 
the  Assistance  Program  is  to  provide  a service,  at  no  cost 
to  the  user,  serious  consideration  should  be  given  to  being 
content  to  see  the  value  of  ROI  remain  above  1.0,  at  which 
point  the  cost  of  providing  the  service  is  offset  by  benefits 
to  its  users. 

The  benefits  of  the  Assistance  Program  are  many  and  varied. 
People  in  the  field,  at  shore  activities  around  the  world 
receive  the  primary  benefits.  However,  engineers  and  scien- 
tists at  CEL  share  in  the  benefits  of  the  Assistance  Program 
too. 

Although  the  effectiveness  of  the  program  will  be  stated 
in  terms  of  estimated  ROI,  it  should  be  noted  that  these  data 
are  based  only  on  that  part  of  the  total  benefit  that  can  be 
quantified  with  reasonable  accuracy  and  credibility.  Since 
a variety  of  additional  benefits  result  from  assistance 
efforts,  it  is  worthwhile  to  list  some  of  them,  so  they  will 
not  be  overlooked  until  such  time  that  techniques  become 
available  to  quantify  them.  Some  of  these  "additional  bene- 
fits" are  listed  in  Appendix  B. 


IV.  METHODOLOGY 


A.  SELECTION  OF  AN  OPE  RAT IONA!  SYSTEM  TO  MEASURE 

EFFECTIVENESS 

Since  its  inception,  the  FESO  has  been  interested  in  the 
evaluation  of  benefits  as  a measure  of  effectiveness,  for  use 
by  CEL  and  NAVFAC . Efforts  were  undertaken  at  the  Naval  Post 
graduate  School  to  measure  the  effectiveness  of  the  FY-72  and 
FY-73  assistance  efforts  [Refs.  6 and  7].  These  efforts  pro- 
duced a basic  foundation  for  later  efforts,  although  the 
results  were  not  definitive  in  that  they  were  based  on  less 
than  adequate  data  gathering  instruments , or  questionnaires 
for  gathering  evaluation  data,  and  insufficient  coverage  of 
the  shore  activities  served.  Later  efforts  by  Hendrickson 
and  Fisher  [Ref.  8],  to  measure  the  effectiveness  of  the  FY- 
74  Assistance  Program,  produced  a more  comprehensive  question 
naire  and  increased  the  evaluation  efforts  to  include  a 
larger  postion  of  the  activities  served.  A breakthrough  also 
resulted  in  that  the  techniques  for  evaluating  the  worth  of 
tangible  and  intangible  benefits  were  advanced  considerably. 
Although  the  study  produced  a workable  system,  the  cost  to 
measure  program  effectiveness  on  an  on-going  basis,  using 
those  techniques  to  evaluate  each  assistance  request  received 
by  CEL,  was  considered  prohibitively  expensive  in  terms  of 
available  resources  and  in  relation  to  the  total  cost  of  pro- 
viding assistance.  Study  results,  which  indicated  a mean 
benefit  of  $2.72  for  each  dollar  expended  to  provide  assist- 
ance during  FY-74,  coupled  with  the  potential  cost  of  similar 
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measurement  efforts,  did  relieve  the  immediate  pressure  for 
continuing  efforts  to  measure  return  on  investment  for  the 
program.  This  led  to  a decision  in  FY-75  to  defer  further 
effectiveness  measuring  for  several  years.  The  shift  in  fund- 
ing source  to  NAVFAC  mission  management  funds  created  need 
for  an  operable  system.  It  was  believed  that  an  operational 
system  incorporating  those  techniques  could  be  developed 
based  on  a modification  of  the  Hendrickson  and  Fisher  model. 
The  cost  barrier  associated  with  evaluating  all  assistance 
requests  received  during  a given  period  could  be  overcome  by 
evaluating  the  benefit  of  a selected  number  of  the  requests 
and  using  these  data  as  a basis  for  estimating  the  total  bene- 
fit of  all  requests. 

B.  SELECTION  OF  A PERIOD  TO  STUDY 

During  FY-76,  CEL  responded  to  612  individual  assistance 
requests  from  Naval  shore  activities.  Although  some  thought 
was  originally  given  to  including  all  those  requests  in  the 
study,  a review  of  the  assistance  activity  during  that  year 
revealed  several  reasons  why  the  study  should  focus  on  the 
last  quarter  of  FY-76.  The  main  reasons  for  selecting  that 
period  for  study  purposes  were: 

1.  The  total  of  145  assistance  requests  during  the  fourth- 
quarter  was  approximately  one-fourth  of  the  total  number 

of  requests  received  during  the  entire  year. 

2.  The  total  of  145  requests  was  approximately  one-half  of 
the  number  of  requests  treated  in  the  Hendrickson  and  Fisher 
study  of  FY-74  assistance  efforts. 

3.  The  145  requests  should  represent  a large  enough  samp- 
ling population  for  study  purposes. 
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C.  SYSTEM  MODEL 

The  system  model  is  based  on  the  evaluation  of  individual 
benefits  to  users.  Although  benefit  is  considered  on  a 
request-by-request  basis,  in  the  final  analysis  the  model 
treats  the  entire  Assistance  Program  as  if  it  were  a single 
CEL  product  resulting  from  a single  investment,  rather  than 
a number  of  individual  requests  for  assistance  with  their 
attendant  costs  and  benefits.  This  follows  the  character  of 
the  R&D  process,  in  that  successes  should  make  up  for  failures. 
Without  failures  progress  will  generally  be  slow  because  the 
approach  will  have  been  too  conservative. 

The  model  provides  for  input  from  the  providers  and  users 
of  assistance  services  and  for  administration  and  coordination 
by  the  FESO.  The  reasoning  behind  these  features  is  discussed 
in  Appendix  C . 

Specifically,  the  model  recognizes  that  the  investment 
required  to  respond  to  some  assistance  requests  will  not  pro- 
vide a measurable,  commensurate  return.  This  is  not  say  that 
satisfying  a particular  request  does  not  provide  benefit,  for 
satisfaction  itself  is  a form  of  benefit.  However,  it  is 
convenient  from  an  operational  viewpoint,  and  less  discon- 
certing to  all  concerned,  to  classify  those  requests  having 
marginal  or  intangible  benefit  as  providing  immeasurable  or 
"zero"  benefit,  when  measuring  effectiveness. 

No  attempt  is  made  to  apportion  credit  to  individual 
requests,  or  among  the  various  requesting  and  responding 
groups.  In  this  way,  the  overall  rewards  for  participation 
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which  would  attend  being  involved  in  a request  which  was  later 
shown  to  be  of  marginal  value.  In  short,  there  was  a conscious 
effort  to  avoid  any  appearance  of  penalizing  the  participants, 
whether  they  are  located  at  CEL  or  at  shore  activities. 

It  is  believed  that  this  system  will  provide  information 
NAVFAC  Code  03  needs,  to  know  whether  it  is  doing  the  right 
thing  in  funding  the  Assistance  Program  with  mission  manage- 
ment funds. 

Two  questions  can  be  asked  in  relation  to  justifying  the 
investment  of  funds  in  the  Assistance  Program.  These  ques- 
tions are: 

1.  Did  the  Assistance  Program  produce  an  acceptable  level 
of  benefit? 

2.  What  level  of  benefit  was  achieved  by  the  Assistance 
Program? 

The  answers  to  both  of  these  questions  are  valuable,  but 
each  carries  a different  cost  in  terms  of  effort  and  timeli- 
ness of  availability.  Efforts  to  determine  whether  the  Assist- 
ance Program  produced  an  acceptable  level  of  benefit  during 
a given  fiscal  year  could  be  carried  out  and  completed  during 
that  fiscal  year.  Results  would  be  available  to  management 
in  time  to  influence  decisions  concerning  program  efforts  in 
the  following  fiscal  years.  By  contrast,  evaluation  efforts 
to  determine  the  overall  level  of  benefit  for  a given  fiscal 
year  would  require  more  effort  and  could  not  be  completed 
until  after  the  end  of  that  fiscal  year.  Thus  valuable  man- 
agement information  would  not  be  available  until  after  final 
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decisions  had  been  made  on  funding  of  the  next  fiscal  year's 
efforts.  Also  management  pressure  for  this  information  could 
be  expected  to  result  in  a period  of  concentrated  evaluation 
effort  shortly  after  the  end  of  the  fiscal  year. 

From  an  operational  viewpoint,  there  was  strong  intuitive 
appeal  for  answering  the  first  question.  Yet,  it  is  believed 
that  effectiveness  information  on  the  overall  level  of  benefit 
would  be  more  useful,  and  therefore  of  more  interest  to  man- 
agement. In  view  of  the  importance  of  these  two  divergent 
views,  an  attempt  was  made  to  satisfy  both  of  them. 

D.  EXPANDING  THE  FESO  AUTOMATED  INFORMATION  SYSTEM 

The  feasibility  of  adapting  a computer  software  package 
for  a small  service  organization  had  already  been  demonstrated 
by  Suess  and  Thaler  [Ref.  9].  A computerized  information  sys- 
tem had  been  developed  and  demonstrated  for  the  FESO  using 
the  Statistical  Package  for  Social  Sciences  (SPSS) . This  sys- 
tem is  currently  being  used  by  the  FESO,  a testimonial  to  the 
value  of  that  effort. 

Suess  and  Thaler  stressed  the  importance  of  adequate  sup- 
porting documentation  in  developing  the  information  system. 

It  was  considered  no  less  important  when  modifying  the  exist- 
ing system,  in  that  it  would  enable  the  FESO  to  understand 
and  effectively  maintain  the  automated  information  system 
using  the  SPSS.  Accordingly,  detailed  documentation  of  the 
processing  logic  and  operating  procedures  for  expanding  the 
FESO  Automated  Information  System  to  process  information  on 
benefit  was  included  as  part  of  this  study.  The  expansion  of 
this  system  is  discussed  in  Section  VI. 
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V.  FEASIBILITY  OF  MEASURING  EFFECTIVENESS  BY  SAMPLING 


A.  OVERVIEW 

The  answer  to  the  question  of  whether  the  Assistance  Pro- 
gram produced  an  acceptable  level  of  benefit  during  the 
fourth  quarter  of  FY-76,  when  145  requests  were  received, 
could  be  obtained  by  a somewhat  simplistic  sampling  procedure 
which  had  considerable  operation  appeal.  By  contrast,  the 
answer  to  the  question  of  what  level  of  benefit  was  achieved 
during  that  period  would  require  application  of  a more  rigor- 
ous procedure.  Both  procedures  were  pursued.  Either  of  them, 
if  shown  to  produce  reasonably  accurate  estimates  of  benefit, 
would  provide  NAVFAC  Code  03  with  on-going  information  on 
program  effectiveness  within  available  CEL  resources. 

This  section  describes  the  efforts  to  develop  and  demon- 
strate an  operational  system  to  obtain  adequate  information 
on  effectiveness  measures  with  an  affordable  level  of  effort. 
Because  of  its  detailed  nature,  the  results  obtained  from  the 
various  approaches  are  previewed  in  this  subsection,  with  the 
details  included  in  subsections  to  follow. 

The  original  approach  was  limited  to  determining  whether 
the  Assistance  Program  had  produced  an  acceptable  level  of 
benefit.  The  initial  procedure  was  to  subjectively  rank  each 
request  in  terms  of  ranges  of  estimated  potential  benefit. 

Then  the  evaluation  of  individual  requests  with  potentially 
large  benefits  would  proceed  and  continue  until  the  aggregated 
benefit  exceeded  the  total  cost  of  providing  assistance.  At 
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that  point  evaluation  efforts  would  stop.  Although  these 
procedures  were  not  followed  exactly,  in  that  efforts  were 
not  concentrated  on  evaluating  potentially  large  benefits, 
it  was  shown  that  the  benefits  of  ten  evaluated  requests,  or 
approximately  seven  percent  of  the  total  of  145  requests, 
exceed  the  $53,766  cost  of  the  program. 

The  direction  of  the  study  was  then  changed  to  an  approach 
which  measured  effectiveness  in  terms  of  ROI,  based  on  the 
evaluation  of  requests  in  randomly  sampled  groups  of  requests. 
The  sampling  population  was  reduced  to  115  requests  by  elim- 
inating 25  requests  subjectively  ranked  as  producing  "zero" 
benefit  and  five  duplicate  requests.  Three  trials  were  made 
using  a multiple  group  sampling  procedure  with  three,  four, 
and  then  five  groups  of  four  requests  each.  Ranges  of  total 
estimated  benefit  were  then  calculated  assuming  a 95%  confi- 
dence interval.  They  were  too  wide.  In  addition,  the  lower 
limit  was  negative  and  therefore  unacceptable.  The  range 
narrowed  and  converged  toward  acceptable  limits  with  each  suc- 
ceeding trial.  This  was  caused  by  the  small  number  of  sample 
groups  and  the  wide  range  of  individual  benefits.  This  samp- 
ling procedure  was  abandoned  before  the  lower  limit  of  the 
range  of  estimated  total  benefit  assumed  a positive  value, 
although  the  benefits  of  20  requests,  or  17  percent  of  the 
total  requests,  had  been  included  in  the  calculations. 

The  random  sampling  approach  was  then  changed  to  include 
a procedure  that  parallels  attribute  inspection  in  the  manu- 
facturing industry.  Six  additional  trials  were  made  using  a 
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grouped  sequential  sampling  procedure.  The  group  size  was 
progressively  increased  to  include  4,  8,  12,  15,  18,  and  then 
45  requests.  Ranges  of  estimated  total  benefit  were  again 
produced,  which  were  consistently  narrower  than  those  produced 
by  multiple  group  sampling.  An  acceptable  range  was  achieved 
with  the  second  additional  trial  (No.  5)  which  included  eight 
requests,  or  seven  percent  of  the  total.  The  range  was  still 
quite  wide,  but  narrowed  with  each  succeeding  trial.  Results 
based  on  the  benefits  of  45  evaluated  requests  indicated  that 
the  minimum  acceptable  estimated  ROI  for  the  Assistance  Pro- 
gram during  the  fourth  quarter  of  FY-76  would  be  realized 
within  a range  between  seven  and  24,  with  a confidence  of  95 
percent . 

In  view  of  the  concern  for  operational  simplicity,  the 
potential  for  expanding  the  original  subjective  ranking  approach 
to  produce  estimates  of  total  benefit  was  then  explored.  Using 
this  approach,  the  total  benefit  was  estimated  to  be  $707,017, 
which  corresponded  to  a return  on  investment  of  13. 

It  was  also  shown  that  the  Assistance  Program  had  produced 
an  acceptable  level  of  benefit.  Identified  benefits  of  the  45 
evaluated  requests  totaled  $324,017,  or  a return  on  investment 
of  six. 

B.  SELECTION  OF  EVALUATION  TECHNIQUES 

Before  the  study  had  progressed  vary  far,  the  decision  was 
made  to  have  the  EFD  RDT&E  liaison  engineers  evaluate  the 
requests  from  their  organizations  as  had  been  done  xn  the  FY- 
72,  FY-73,  and  FY-74  surveys.  They  were  requested  to 
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coordinate  the  evaluation  of  those  requests  with  the  author 
administering  the  evaluation  of  requests  from  other  organiza- 
tions. A modified  questionnaire  was  developed  and  used  to 
obtain  evaluation  data  from  field  users.  The  development  of 
this  questionnaire  is  discussed  in  Appendix  D. 

C.  SUBJECTIVE  RANKING 

A review  of  the  data  from  the  three  previous  surveys 
revealed  a wide  range  of  individual  benefit  values  with  a 
small  number,  about  five  percent,  of  relatively  large  benefit 
values  included  among  them.  This  review  provided  the  basis 
for  the  original  approach,  which  was  to  determine  only  if  the 
Assistance  Program  had  produced  an  acceptable  level  of  bene- 
fit, i.e.,  a return  on  investment  of  1.0.  Some  other  minimum 
value  of  ROI  could  be  selected,  for  this  value  defines  what 
is  considered  an  acceptable  level  of  benefit,  but  recovery 
of  cost  was  considered  to  be  a minimum  requirement. 

Each  of  the  145  requests  were  subjectively  ranked  in  terms 
of  their  estimated  potential  benefit  using  the  same  benefit 
value  ranges  included  in  the  revised  form  for  recording  verbal 
requests  for  assistance.  (Appendix  E.)  The  results  of  this 
ranking  effort,  to  subjectively  identify  the  "larger"  benefits, 
indicated  that  a total  of  five  requests  were  each  estimated 
to  provide  potential  benefit  in  excess  of  $25,000  (Table  I). 

The  operational  appeal  of  this  approach  was  increased  with 
the  thought  that  evaluating  those  five  requests,  or  less  than 
four  percent  of  the  total  requests,  might  identify  benefits 
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that  amounted  to  more  than  twice  the  cost  ($53,766)  of  pro- 
viding assistance  services  during  the  fourth  quarter  of  FY-76. 

At  the  time  the  evaluation  of  those  five  requests  was 
ready  to  begin,  the  expanded  FESO  Automated  Information  System 
was  ready  to  be  tested  with  live  data.  Rather  than  wait  to 
obtain  data  from  evaluating  the  "larger"  benefit  requests, 
six  requests  which  had  been  evaluated  by  the  EFD  RDT&E  liaison 
engineers  were  chosen  so  as  to  provide  a variety  of  data  for 
test  purposes.  The  total  benefit  from  these  six  requests  was 
determined  to  be  $46,828,  or  87  percent  of  the  cost  of  the 
Assistance  Program. 

As  testing  of  the  expanded  information  system  continued, 
and  again  without  concentrating  on  the  "larger"  benefit 
requests,  the  number  of  evaluated  requests  included  in  the 
data  base  was  increased  from  six  to  ten.  Three  more  requests 
evaluated  by  the  EFD  representatives  were  added,  and  one 
request  evaluated  by  the  author.  These  four  included  one 
request  which  had  been  evaluated  as  a duplicate,  and  therefore 
of  zero  benefit.  The  total  benefit  of  these  ten  requests 
amounted  to  $54,428.  Thus  identified  benefits  from  approxi- 
mately seven  percent  of  the  total  requests,  had  exceeded  the 
cost  of  the  program.  In  addition,  the  identified  benefit  of 
nine  of  these  requests  was  in  the  range  of  potential  benefit 
as  ranked  by  the  author.  The  results  of  these  ten  evaluations 
are  shown  in  Table  II. 

Although  the  results  were  encouraging,  serious  doubts 
were  forming  as  to  their  credibility  and  usefulness  to  man- 
agement. There  would  be  some  who  would  question  the  approach. 
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TABLE  I 

SUBJECTIVE  RANKING  OF  ASSISTANCE  REQUESTS 

Benefit  Estimate  Range  of  Estimated  Number  of  Percentage 
Code  Potential  Benefit  ($)  Requests  of  Total 


1 

0 

25 

17 

2 

0 

- 

499 

19 

13 

3 

500 

- 

4999 

73 

50 

4 

5000 

- 

24999 

18 

12 

5 

Over 

- 

25000 

5 

4 

6 (Duplicate) 

0 

5 

4 

TOTAL 

145 

100 

TABLE  II 

BENEFITS  OF  FIRST  TEN  ASSISTANCE  REQUESTS 


Request  Number 

Benefit  Estimate 
Code 

Benefit  ($) 

First  Six 

557 

1 

0 

643 

4 

23,862 

647 

1 

0 

682 

4 

20,000 

688 

3 

716 

720 

3 

2,258 

Next  Four 

559 

3 

6,000 

587 

6 

0 

588 

3 

0 

3 


715 


3 5 


TOTAL 


1 , 600 
54,428 


Therefore,  the  direction  of  the  study  was  modified  and  the 
scope  expanded  to  overcome  potential  barriers  to  future  use. 

The  direction  was  changed  to  determine  what  level  of  bene- 
fit had  been  achieved,  stated  in  terms  of  ROI. 

D.  RANDOM  SAMPLING  APPROACH 

In  this  approach,  requests  were  randomly  selected,  evalu- 
ated, and  their  individual  benefits  aggregated  and  used  as  a 
basis  for  estimating  the  total  benefit  of  all  requests 
received.  The  estimated  total  benefit  was  then  divided  by 
the  total  investment  in  the  Assistance  Program  to  determine 
an  estimated  ROI.  The  aim  was  to  achieve  significant  results 
with  the  minimum  size  sample,  so  that  evaluation  efforts  in 
an  on-going  system  could  be  maintained  at  a manageable  level. 
Somewhat  arbitrarily,  although  based  on  results  to  that  point 
in  the  study,  a ten  percent  sample  size  was  selected.  With 
a sampling  population  of  115  requests,  the  sample  consisted 
of  12. 

A multiple  group  sampling  procedure  using  three  separate 
sample  groups  of  four  requests  each,  with  replacement  taking 
place  before  each  new  group  was  selected,  was  chosen  as  the 
most  appropriate  procedure  for  this  sample  size.  The  premise 
was  that  the  mean  and  standard  deviation  of  the  sample  group 
means  would  provide  the  best  estimate  of  the  mean  of  the 
population  of  requests  sampled. 

The  SPSS  provides  a procedure  for  taking  a random  sample 
from  a file  (population)  for  processing.  A factor  is  selected 
which  is  a positive  number  less  than  1.0  which  indicates  the 
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For  instance. 


percentage  of  the  file  that  is  to  be  analyzed, 
a factor  of  0.1  would  result  in  approximately  a 10  percent 
sample  for  the  entire  file.  The  probability  of  selecting  any 
particular  request  in  the  file  is  equal  to  the  factor  speci- 
fied. However,  as  each  request  is  considered  for  selection 
independently  of  all  other  requests,  the  resulting  set  of 
sampled  requests  will  generally  not  be  exactly  the  size  speci- 
fied. The  sample  procedure  used  by  SPSS  is  set  up  in  such  a 
way  that  a different  random  sample  will  be  selected  each  time 
the  user  requests  a sample  [Ref.  10]. 

A factor  of  0.033  was  selected  to  obtain  the  three  groups 
of  four  requests  each  for  a series  of  multiple  sampling. 
However,  rather  than  obtaining  this  grouping,  the  SPSS  proce- 
dure consistently  provided  samples  containing  three,  two,  and 
five  requests.  Repeated  attempts  to  obtain  three  or  more 
equal  size  samples  proved  fruitless  and  efforts  to  use  the 
SPSS  sample  procedure  were  abandoned. 

Using  the  random  number  generator  program  in  an  HP-25 
programmable  hand-calculator,  three  groups  of  four  requests 
each  were  selected.  Field  evaluations  via  telephone  inter- 
views were  performed  by  the  author  for  those  selected 
requests  not  already  evaluated  by  the  EFD  RDT&E  liaison 
engineers.  The  resulting  mean  of  the  mean  benefit  of  these 
three  sample  groups  was  $12,134,  with  a standard  deviation 
of  $9,607.  Although  the  benefit  values  of  individual 
requests  are  assumed  to  be  normally  distributed,  the  sampling 
distribution  was  probably  not  normal  since  the  sample  size 


was  small  (less  than  30).  Accordingly,  the  student's  "t" 
distribution  for  small  sized  samples  was  used  [Ref.  11].  The 
95%  confidence  interval  of  the  mean  benefit  was  determined  to 
range  between -$11,  734  and  $36 , 002  . The  corresponding  range 
of  total  benefit  for  the  population  of  115  requests  was  from 
-$1,349,410  to  $4,140,230.  Thus  the  estimated  range  of  ROI 
was  between  -25  and  77  with  a confidence  of  95%. 

A review  of  these  figures  in  the  light  of  the  benefits 
already  identified  spoke  for  the  need  to  seek  a narrower 
confidence  interval. 

Another  set  of  requests  was  randomly  selected  to  comprise 
the  fourth  sample  group.  Again,  missing  evaluations  were  ob- 
tained from  telephone  interviews.  The  sample  then  consisted 
of  16  requests,  divided  into  four  groups.  The  resulting  mean 
of  the  mean  benefit  shifted  to  the  left  and  the  confidence 
interval  was  narrowed  somewhat,  although  it  was  still  not 
definitive  since  the  lower  limit  remained  negative.  The 
resultant  range  of  ROI  was  -11  to  52. 

The  number  of  sample  groups  was  increased  again,  to  five, 
in  an  attempt  to  gain  further  narrowing.  The  process  of  ran- 
domly selecting  another  group  of  four  requests,  obtaining 
missing  evaluations  and  performing  the  necessary  calculations 
was  repeated  as  above.  The  results  were  encouraging  in  that 
the  range  of  ROI  did  narrow  further,  although  the  left  end 
remained  negative.  The  results  of  these  three  sampling  trials 
are  shown  in  Table  III. 
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TABLE  III 


ESTIMATED  TOTAL  BENEFIT  BY  MULTIPLE  GROUP  SAMPLING 


Limits  of  95%  Confidence  Interval 


LIMIT 


BENEFIT 


X ($)  TOTAL  ($)*  I ROI 


Trial  No.  1 


3 groups  of  4 
requests  each  — 

"x  = $12,234 

'S'  = $9,607 


Lower  -11,374  -1,349,410  ! -25 

i 

Upper  36,002  4,140,230  i 77 


Trial  No.  2 


4 groups  of  4 
requests  each  — 

X = $9,639 

S'  = $9,297 


| Lower 

I 

Upper 


-5,152 


-592,480  ; -11 


24,430  2,809,450  j 52 
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At  this  point  it  was  tentatively  concluded  that  the  diffi- 
culty in  obtaining  a fairly  narrow  and  definitive  range  of 
estimated  total  benefit  was  mainly  due  to  the  small  number 
of  sample  groups  and  the  wide  range  in  value  of  individual 
benefits.  It  was  believed  that  these  complications  would  not 
be  present,  or  at  least  not  to  the  same  degree,  in  on-going 
evaluations  with  a larger  sample  population. 

A similar  situation,  in  which  a wide  range  of  individual 
benefits  is  fixed,  but  the  sample  size  can  be  effectively 
increased,  exists  in  the  manufacturing  industry,  where  inspec- 
tion of  entire  lots  of  goods  is  too  expensive.  Sampling 
inspection  by  attributes  is  used  with  the  intent  of  achieving 
the  desired  type  of  control  information  with  the  smallest 
possible  sample.  In  an  attempt  to  reduce  the  total  amount 
of  inspection  required,  sequential  sampling  is  used.  This 
procedure  consists  of  inspecting  one  item  at  a time,  or  one 
group  of  items  at  a time,  and  after  each  inspection  deter- 
mining whether  the  accumulated  information  from  items  thus 
far  inspected  justifies  a decision  concerning  acceptance  or 
rejection  of  the  lot,  or  whether  another  item  or  group  of 
items  must  be  inspected.  A judgement  concerning  the  whole 
lot  is  made  on  the  basis  of  the  combined  samples  [Refs.  12 
and  13].  In  attribute  inspection  applied  to  manufacturing, 
upper  and  lower  acceptance  limits  are  established  for  defec- 
tive items  as  the  basis  for  an  accept  or  reject  decision. 

In  this  study,  where  the  attributes  are  benefits,  an  "accept- 
able" range  was  achieved  when  the  upper  and  lower  confidence 


40 


limits  were  both  positive.  This  amounts  to  rejecting  the 
situation  where  a large  isolated  benefit  or  a significant 
number  of  zero  benefits  produce  non-definitive  results.  An 
acceptable  range  may  still  be  fairly  wide,  although  evalua- 
tion of  additional  groups  will  narrow  the  range  further. 

Grouped  sequential  sampling  was  chosen  so  that  results 
could  be  compared  with  those  previously  obtained  from  multiple 
group  sampling.  Specifically,  a fourth  trial  was  run  with 
the  evaluated  benefits  of  the  four  requests  in  the  first 
group  of  the  multiple  group  sampling  procedure  used  to  cal- 
culate estimated  total  benefit.  If  the  sample  were  suffici- 
ently convincing,  a decision  could  be  made  about  the  entire 
program  at  that  point.  Previous  results  did  not  indicate 
that  this  would  be  so,  although  it  might  be  in  future  evalu- 
ations. Accordingly  a fifth  and  sixth  trial  group  were 
tested  using  combined  samples  of  eight  and  12  requests.  At 
that  point  duplicate  requests  appeared  in  the  random  sample 
groups  which  had  the  effect  of  reducing  the  combined  samples 
for  the  seventh  and  eighth  trials  to  15  and  18  requests. 

One  more  possible  test  of  this  procedure  existed.  A total 
of  45  requests  had  been  evaluated  at  that  time  by  the  EFD 
RDT&E  liaison  engineers  and  the  author.  These  did  not  include 
any  duplicate  requests  or  those  ranked  in  the  "zero"  benefit 
range,  which  were  excluded  from  the  random  sampling  approach. 
The  ninth  trial  consisted  of  testing  these  45  requests  as 
one  sample  to  see  whether  a narrower  range  of  estimated  total 
benefit  would  result.  This  was  a biased  trial,  although  the 
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inaccuracy  was  believed  to  be  minimal  since  40  percent  of  the 
requests  had  been  randomly  selected.  Also,  since  the  sample 
of  45  requests  represented  39  percent  of  the  total  of  115 
requests  in  the  population,  it  should  produce  a fairly  reli- 
able estimate  of  total  benefit.  The  results  of  the  six  grouped 
sequential  sampling  trials  are  shown  in  Table  IV. 

The  identified  benefits  of  the  18  requests  in  Trial  No.  8 
amounted  to  $145,844  and  the  identified  benefits  of  the  45 
requests  in  Trial  No.  9 totaled  $324,019.  These  values  pro- 
vided sufficient  evidence  to  conclude  that  it  was  feasible 
to  obtain  meaningful  ranges  of  estimated  total  benefit  with  a 
random  sampling  approach,  and  no  further  attempts  were  made 
to  narrow  the  range.  At  that  point  it  had  been  determined 
that  the  estimated  ROI  for  the  Assistance  Program,  during  the 
fourth  quarter  of  FY-76,  would  fall  within  a range  between 
seven  and  24,  with  a confidence  of  95  percent. 

Several  significant  differences  were  apparent  from  a com- 
parison of  the  results  from  grouped  sequential  sampling  with 
those  from  multiple  group  sampling: 

1.  The  range  of  estimated  total  benefit  was  wider  with 
multiple  group  sampling,  and  although  it  narrowed  faster  and 
more  uniformly  than  with  grouped  sequential  sampling,  the 
lower  limit  remained  negative  after  benefits  of  16  percent 
of  the  requests  had  been  included  in  the  samples. 

2.  The  range  of  estimated  total  benefit  was  consistently 
narrower  with  grouped  sequential  sampling,  and  the  lower  limit 
became  positive  by  the  time  benefits  of  seven  percent  of  the 
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TABLE  IV 


ESTIMATED  TOTAL  BENE: 'IT  BY  GROUPED  SEQUENTIAL  SAMPLING 


Limits  of  95%  Confidence  Interval 


LIMIT 


BENEFIT 


X ($)  TOTAL  ($)*  ROI 


Trial  No.  4 


4 requests  — 
X = $2,383 

S'  = $4,035 


Lower 


Upper 


-3,218  i -370,116 


7,983 


918,045 


Trial  No.  5 


8 requests  — 
~X  = $11,987 
S'  = $11,482 


Lower  2,626 


301,967 


Upper  21,347  2,454,951 


12  requests  — 


Lower 


Upper 


3,682 


423,419 


20,586  2,367,355 


* for  115  requests 


where  X = mean  benefit 


S = sample  standard  deviation 
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L'  r 


2,618 


301,070 


Upper  I 16,828  j 1,935,220  | 36 


TABLE  IV  (Continued) 

ESTIMATED  TOTAL  BENEFIT  BY  GROUPED  SEQUENTIAL  SAMPLING 

Limits  of  95%  Confidence  Interval 

BENEFIT 

LIMIT  — — 

X ($)  TOTAL  ($)*  ROI 

Trial  No.  7 

15  requests  — 

7 = $9,723  Lr  ^r  2,618  301,070  6 

's  = $12,914  Upper  16,828  1,935,220  36 


Trial  No.  8 

18  requests  — 

7 = $8,102 

S'  = $12,298 


Trial  No.  9 

45  request  — 
7 = $7,200 

S'  = $14,183 


Lower 

Upper 

I 

1,986 

14,218 

Lower 

Upper 

1 

3,056 

11,344 

228,424 


351,440 


* for  115  requests 


where  X = mean  benefit 

S = sample  standard  deviation 
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requests  had  been  included  in  the  sample  group,  although  the 
range  was  still  quite  wide. 

The  results  of  the  nine  random  sampling  trials  are  shown 
graphically  in  Figure  4 for  comparative  purposes.  Since  the 
calculations  of  statistics  were  performed  external  to  SPSS, 
using  an  HP-25  programmable  calculator,  the  procedures  were 
not  included  in  the  modified  FESO  Automated  Information  Sys- 
tem. However,  typical  calculations  are  included  in  Appendix  F. 

E.  ESTIMATING  APPROACH 

With  the  knowledge  that  a random  sampling  approach  would 
produce  meaningful  benefit  information  for  management  use,  the 
possible  value  of  subjectively  ranking  requests  into  ranges 
of  estimated  potential  benefit  was  explored  further.  By 
multiplying  the  number  of  requests  in  each  range  by  the  mid- 
value of  that  range,  and  summing  these  range  totals,  an  esti- 
mate of  total  benefit  could  be  obtained.  Since  the  mid-value 
of  the  "over  $25,000"  range  was  unknown,  the  value  of  $25,000 
was  arbitrarily  assumed.  Performing  these  calculations 
resulted  in  an  estimated  total  benefit  of  $600,500  (Table  V). 
The  value  was  well  situated  within  the  95  percent  confidence 
limits  of  from  $351,440  to  $1,304,560  obtained  in  sampling 
Trial  No.  9.  Actually,  it  was  conservative  as  it  was  38  per- 
cent below  the  mid-point  value  of  $828,000. 

An  opportunity  existed  to  see  how  conservative  the  benefit 
estimating  approach  was,  by  comparing  it  with  the  identified 
benefits  of  the  45  requests  included  in  Trial  No.  9.  Using 
those  45  requests,  the  procedure  described  above  was  repeated. 


The  estimated  total  benefit  was  calculated  to  be  $263,508, 
compared  to  the  identified  total  benefit  of  $324,017.  The 
estimating  approach  was  conservative  by  19  percent. 

A review  of  the  individual  benefits  of  the  45  requests 
showed  that  three  requests  produced  benefits  greater  than 
$25,000.  The  values  of  these  benefits  were  $75,165,  $39,770, 
and  $30,000.  Their  mean  value  was  $48,312.  Using  $48,000 
for  the  mid- value  of  the  "over  $25,000"  range  should  increase 
the  reliability  of  the  estimate  since  it  would  then  be  based 
on  actual  benefit  data.  Recalculating  the  total  benefit 
using  this  revised  mid-value  resulted  in  an  estimated  value 
of  $715,500.  This  value  was  closer  to  the  mid-point  value  of 
$828,000  from  Trial  No.  9,  but  still  14  percent  below  it. 

Using  $48,000  for  the  mid-value  of  the  "largest"  range 
and  repeating  the  comparison  with  the  45  evaluated  requests  in 
Trial  No.  9 resulted  in  an  estimated  total  benefit  of  $332,500. 

A difference  of  only  three  percent  from  the  identified  total 
benefit  of  $324,017  was  accepted  as  an  indication  that  the 
estimating  approach,  as  practiced  by  the  author,  had  been  suc- 
cessfully demonstrated  to  be  reasonably  accurate. 

After  that  successful  demonstration,  an  estimate  of  total 
benefit  was  made  based  on  the  actual  benefit  that  had  been 
identified  and  the  estimated  benefit  remaining  in  the  70 

■ 

requests  that  had  not  been  evaluated.  The  results  of  these 
calculations  indicated  an  estimated  total  benefit  of  $707,017 
(Table  VI) . Although  this  value  was  very  close  to  the  origi- 
nal estimated  value  of  $715,500,  it  was  not  taken  as  an 
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TABLE  V 


ORIGINAL 

ESTIMATE  OF  TOTAL  BENEFIT 

Range  of  Estimated 
Potential  Benefit  ($) 

Mid- value 
of  Range  ($) 

Number  of 
Requests 

Estimated 
Benefit  ($) 

0 

0 

30 

0 

0 - 499 

250 

19 

4750 

500  - 4999 

2750 

73 

200750 

5000  - 24999 

15000 

18 

270000 

over  25000 

25000 

5 

125000 

TOTAL 

145 

600500 

TABLE  VI 

ESTIMATE  OF  TOTAL 

BENEFIT  BASED 

ON  EVALUATED 

BENEFIT 

Range  of  Estimated 

Mid-value 

Number  of 

Estimated 

Potential  Benefit  ($) 

of  range 

Requests 

Benefit  ($ 

0 0 30  0 


0 - 

499 

250 

9 

2250 

500  - 

4999 

2750 

49 

134750 

5000  - 

24999 

15000 

10 

150000 

over 

25000 

48000 

2 

96000 

SUBTOTALS 

70 

383000 

EVALUATED 

REQUESTS 

45 

324017 

TOTALS 

115 

707,017 
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indication  that  these  estimates  of  total  benefit  were  any 
better  than  those  obtained  by  the  grouped  sequential  sampling 
procedure.  What  was  apparent,  was  that  the  estimating  approach 
provided  another  valuable  tool  for  the  FESO  to  use  in  estim- 
ating total  benefit  with  reasonably  accuracy.  For  this  rea- 
son, procedures  were  incorporated  into  the  expanded  FESO 
Automated  Information  System  to  facilitate  these  calculations. 
These  procedures,  along  with  other  system  modifications  are 
discussed  in  section  VI . 
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VI.  EXPANSION  OF  THE  FESO  AUTOMATED  INFORMATION  SYSTEM 


Using  the  Statistical  Package  for  the  Social  Sciences 
(SPSS) , an  automated  information  system  was  developed  and 
demonstrated  by  Suess  and  Thaler  [Ref.  9].  It  has  since  been 
implemented  to  satisfy  the  data  processing  requirements  of 
the  FESO  using  the  Lawrence  Berkeley  Computing  Center  via  a 
batch  terminal. 

Since  the  gathering  and  processing  of  information  on  the 
benefits  of  assistance  requests  will  be  an  on-going  require- 
ment for  the  FESO,  it  was  a logical  next  step  to  accommodate 
by  expanding  the  FESO  Automated  Information  System.  The 
expansion  of  the  system  to  satisfy  the  information  require- 
ments of  an  effectiveness  measuring  system  is  discussed  in 
the  remainder  of  this  section. 

A.  STATEMENT  OF  SYSTEM  OBJECTIVES 

The  major  objectives  in  expanding  the  FESO  Automated 
Information  System  are: 

1.  To  relieve  the  FESO  of  as  much  time-consuming  manual 
data  manipulation  as  possible. 

2.  To  provide  a method  to  process  benefit  information  on 
any  or  all  requests  received  during  any  or  all  quarters  of 
a fiscal  year. 

3.  To  provide  a method  to  verify  automatically  the 
accuracy  of  the  data  input  to  the  computer. 

4.  To  accurately  process  and  generate  all  of  the  benefit 
information  required  by  the  FESO  in  preparing  effective- 
ness reports. 
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B.  IDENTIFY  OUTPUT  REPORTS 


The  initial  step  in  expanding  the  system  was  to  identify 
required  output  reports.  To  ensure  adequate  identification 
for  control  purposes,  each  output  report  was  given  a descrip- 
tive title  and  assigned  a report  number. 

Previous  efforts  had  identified  eleven  reports  as  being 
necessary  to  provide  the  FESO  with  the  information  to  function 
effectively.  Eleven  additional  reports  relating  to  benefit 
information  were  identified  as  being  necessary  and  were  given 
individual  designators.  An  inventory  of  these  reports  is  in- 
cluded in  Figure  5. 

C.  PREPARE  REPORT  SPECIFICATIONS 

After  the  output  reports  had  been  identified,  input  and 
data  manipulation  required  to  produce  the  output  reports  were 
determined.  This  was  done  by  preparing  a Report  Specification 
for  each  output  report  listed  in  the  Inventory  of  Reports. 
Preparation  of  the  Report  Specifications  required  a detailed 
systematic  analysis  of  each  output  report  to  determine  the 
data  selection  criteria,  data  elements,  data  manipulation 
logic  and  output  parameters.  The  report  specifications  for 
the  eleven  benefit  reports  are  included  in  Appendix  G.  The 
information  contained  in  the  report  specifications  of  the 
benefit  portion  of  the  information  system  was  used  to  prepare 
the  SPSS  control  cards  and  operating  procedures  described 
later . 
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INVENTORY  OF  BENEFIT  REPORTS 


Report  No. 

OB  Benefit  Edit  Report 

12  Original  Estimate  of  Total  Benefit 

13  Breakdown  of  Evaluation  Status 

14  Breakdown  of  Field  Evaluation  Data 

15  Comparison  of  Contribution  and  Information 
Provided 


16  Summary  of  Savings  by  Type  of  Savings 

17  Estimated  Minimum  Benefit  of  Evaluated  Requests 

18  Summary  of  Evaluated  Benefits  by  Type  of  Savings 

18A  Statistics  for  Evaluated  Benefits 

19  Summary  of  Benefits  by  Benefit  Code 

20  Estimated  Minimum  Remaining  Benefit  by 
Benefit  Estimate 

21  Comparison  of  Benefit  Estimate  and  Benefit 


Figure  5.  Inventory  of  Benefit  Reports 
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D.  COMPILE  DATA  ELEMENT  LIST  AND  ASSIGN  CODES 


With  the  report  specifications  prepared,  the  requirements 
for  the  data  input  became  apparent.  A review  of  the  report 
specifications  provided  a composite  list  of  all  data  elements 
that  should  be  included  as  input  to  the  preparation  of  the 
reports.  A list  of  all  data  elements  used  in  the  benefit 
portion  of  the  information  system  is  shown  in  Figure  6. 

E.  DESIGN  INPUT  RECORD  FORMAT  AND  CODE  DATA  DEFINITION 

The  structuring  of  the  data  elements  followed  from  the 
format  of  the  questionnaire  (input  record) . Figure  7 shows 
the  format  of  the  input  record  for  assistance  benefit  inform- 
ation. Fields  1 through  4 comprise  the  control  group  and  are 
identical  to  those  previously  used,  as  in  Field  5.  Field  5 
is  a special  type  field.  It  is  a redefinition  or  combination 
of  fields  3 and  4.  Fixed  data  comprise  Fields  6 through  15 
and  20  through  22.  Fields  16  through  19  and  23  comprise  the 
variable  group. 

With  the  input  record  format  described,  the  additional 
data  base  was  described  in  computer  language  utilizing  the 
data  definition  language  feature  of  the  SPSS.  This  process 
entailed  defining  the  data  elements  and  their  codes  in  the 
format  specified  in  the  SPSS  Manual.  The  results  of  the  prep- 
aration of  data  definition  cards  describing  the  expanded 
portion  of  the  Assistance  Program  data  base  are  listed  in 
Appendix  H.  Cards  which  relate  to  the  benefit  portion  of 
the  data  base,  or  required  modifications  to  accommodate  it, 
are  preceded  by  a solid  ball  (•) . 
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DATA  ELEMENT  LIST 


Number  of 


Data  Element 

Characters 

Format 

Evaluation 

1 

Numeric 

Estimate  of  Benefit 

1 

Numeric 

Any  Benefit 

1 

Numeric 

No  Benefit 

1 

Numeric 

Benefit  Code 

2 

Numeric 

Use  of  Advice  or  Assist  mce 

2 

Numeric 

Use  Factor 

4 

Numeric 

Discounted  Present  Value  Factor 

5 

Numeric 

Type  of  Savings 

1 

Numeric 

One-Time  Construction  or  Repair 
Cost  Savings 

6 

Numeric 

Annual  Maintenance  Cost  Savings 

6 

Numeric 

Annual  Operating  Cost  Savings 

6 

Numeric 

Project  Cost 

7 

Numeric 

Contribution  of  Advice  or 
Assistance  to  Decision 

1 

Numeric 

Percent  of  Information  Provided 
by  CEL  Compared  to  Total 

3 

Numeric 

Would  Solution  be  Applicable  to 
Other  Activities 

1 

Numeric 

Benefit 

6 

Numeric 

Figure  6.  Data  Element  List 
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INPUT  RECORD  FORMAT  FOR  BENEFIT  DATA 
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F.  CONVERT  REPORT  SPECIFICATIONS  INTO  COMPUTER  TASK 

DEFINITIONS 

A set  of  task-definition  cards  was  then  prepared  to 
describe  the  data  manipulation  and  the  statistical  calcula- 
tions to  be  performed  on  selected  data  for  each  report  to  be 
produced.  The  data-def inition  cards  remain  unchanged  while 
preparing  all  reports.  However,  a set  of  task-definition 
cards  is  required  for  each  report.  Each  set  of  cards  was 
identified  by  using  a TASK  NAME  statement  card. 

The  report  specification  provided  the  information  neces- 
sary to  prepare  the  task-definition  cards.  The  data  selection 
constraints  defined  which  data  should  be  used.  The  data 
manipulation  requirements  defined  which  calculations  were  to 
be  made.  The  task-definition  logic  for  each  of  the  benefit 
information  reports  developed  for  the  Assistance  Program  are 
contained  in  Appendix  I. 

G.  DEVELOP  EDIT  LOGIC 

In  developing  the  editing  logic  for  the  benefit  portion 
of  the  Assistance  Program  data  base,  "sight"  verification  of 
the  input  data  was  implemented  through  the  use  of  the  activity 
file,  a computer  printout  of  all  input  data. 

Prior  to  the  processing  of  any  reports,  the  input  data 
was  run  through  an  edit  operation.  This  edit  operation  was 
designed  by  the  preparation  of  an  additional  set  of  task- 
definition  statements.  This  set  of  task-definition  statements 
checks  each  data  record  for  accuracy  in  conformance  with  the 
coded  data  previously  established  on  the  data-def inition 
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cards  in  the  computer  program.  Figure  8 lists  the  set  of 
task-definition  statements  prepared  for  the  edit  operation. 

The  edit  logic  checks  each  field  in  the  input  record  for 
accurate  and  complete  data  in  two  manners.  First,  the  con- 
tent of  each  field  is  checked  against  the  data-def inition 
section  statements  that  defined  the  structured  data  base  and 
accepted  values.  Each  field  must  have  an  acceptable  value 
or  it  is  flagged  as  an  error.  Next  the  relationship  between 
fields  is  defined  and  the  content  of  the  related  fields  must 
be  consistent.  For  example,  if  for  a particular  request,  the 
type  of  saving  field  has  been  coded  "1,"  indicating  that  a 
one-time  construction  or  repair  savings  has  been  realized, 
and  there  is  no  data  in  the  one-time  savings  field,  the  edit 
logic  will  flag  an  error.  If  the  record  passes  the  edit  with- 
out any  error  flags,  the  data  are  considered  accurate  and 
complete. 

Error  information  about  the  data  record  that  failed  the 
edit  test  is  displayed  through  the  use  of  an  exception  report. 
Error  flags  showing  the  input  card  columns  containing  erron- 
eous data  are  displayed  in  the  report  along  with  the  necessary 
record  identification  information.  When  the  erroneous  data 
card  records  are  corrected  and  all  input  data  passes  the  edit 
with  a "negative"  exception  report  (no  errors) , processing  of 
the  reports  may  begin. 

Artificial  test  data  were  contrived  for  the  benefit  portion 
of  the  information  system  to  test  the  edit  logic.  Invalid 
data  were  constructed  to  test  the  comprehensiveness  of  the 
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edit  logic.  After  successfully  passing  this  test,  representa- 
tive live  data  were  prepared  and  used  as  input  to  test  each 
report  separately.  After  each  report  processing  logic  was 
tested  separately,  a system  test  was  conducted  using  live 
data  and  generating  all  benefit  reports. 

The  test  reports  prepared  for  the  benefit  portion  of  the 
Assistance  Program  data  base  provided  some  interesting  results 
which  were  discussed  in  section  V.  To  illustrate  a typical 
test  report,  the  test  report  used  as  a basis  for  Table  II  is 
shown  in  Figure  9.  The  reason  that  Table  II  lists  ten 
requests,  while  Figure  9 lists  only  seven,  is  that  three  of 
the  requests  were  ranked  as  "zero"  benefit  requests  which  ex- 
cluded them  from  field  evaluations. 
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Figure  9 Test  Report  - Summary  of  Evaluated  Benefits  By  Type  of  Savings  - Report  #18 


VII.  CONCLUSIONS  AND  RECOMMENDATIONS 

A.  CONCLUSIONS 

The  arguments  for  and  the  demonstration  of  the  use  of  two 
conceptually  different  approaches  to  satisfy  NAVFAC  manage- 
ment requirements  to  measure  the  effectiveness  of  a small 
service  program  have  been  presented  in  this  study.  The  follow- 
ing conclusions  have  been  drawn. 

1.  The  data  resulting  from  user  evaluations  of  randomly 
selected  assistance  requests,  when  processed  by  a computer- 
aided  information  system  is  a viable  and  economically  afford- 
able basis  for  estimating  the  return  on  investment  of  the 
resources  expended  on  the  Assistance  Program. 

2.  Grouped  sequential  sampling  procedures  produced  mean- 
ingful ranges  of  estimated  total  benefit  for  requests  received 
during  the  fourth  quarter  of  FY-76.  The  total  benefit  for 

that  period  was  estimated  to  be  between  $350,000  and  $1,300,000, 
which  corresponded  to  a return  on  investment  between  seven  and 
24,  with  a confidence  of  95  percent. 

3.  The  Assistance  Program  produced  an  acceptable  level 

of  benefit  during  the  fourth  quarter  of  FY-76.  Benefits  quan- 
tified in  this  study  amounted  to  $324,017,  or  a return  on 
investment  of  six. 

4.  An  estimating  approach  based  on  subjectively  ranking 
requests  into  ranges  of  potential  benefit  is  a valuable  tool 
for  the  FESO  to  use  in  estimating  total  benefit  of  the  Assist- 
ance Program  with  reasonable  accuracy.  Using  this  approach, 
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the  total  benefit  of  the  Assistance  Program  during  the  fourth 
quarter  of  FY-76  was  estimated  to  be  $710,000,  or  a return  on 
investment  of  13. 


B.  RECOMMENDATIONS 

1.  Each  request  should  be  subjectively  ranked  by  the 
FESO  into  one  of  two  categories;  those  containing  "zero" 
benefit,  and  those  with  potentially  measurable  benefit.  "Zero" 
benefit  requests  should  be  excluded  from  further  evaluation. 

2.  An  effectiveness  measuring  system,  using  a random 
sampling  approach  as  presented  in  this  study,  should  be 
adopted  by  the  FESO  for  measuring  the  effectiveness  of  the 
Assistance  Program  during  FY-77. 

3.  Evaluation  data  for  selected  requests  should  be  ob- 
tained from  field  users,  using  the  feedback  questionnaire 
developed  as  part  of  this  study.  Requests  originating  from 
the  EFDs  should  be  administered  by  the  EFD  RDT&E  liaison 
engineers.  The  FESO  should  administer  the  evaluation  of  all 
other  selected  requests. 

4.  Four  groups  of  four  requests  each  should  be  selected 
from  each  of  the  four  quarters  of  FY-77,  which  should  approxi- 
mate a ten  percent  sample.  Evaluation  data  from  these 
requests  should  be  used  to  calculate  estimated  total  benefit 
and  return  on  investment  using  procedures  for  multiple  group 
sampling,  grouped  sequential  sampling,  and  then  request-by- 
request sequential  sampling  as  the  basis  for  selecting  a single 
sampling  plan  and  an  appropriate  sample  size  for  on-going 
effectiveness  measurements. 
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5.  Additional  work  should  be  undertaken  by  the  FESO  to 
locate  or  develop  procedures  for  selecting  sample  groups  and 
programs  to  perform  the  necessary  statistical  calculations 
for  the  selected  sampling  procedure  which  are  compatible  with 
the  SPSS,  or  can  be  incorporated  into  the  FESO  Automated 
Information  System. 
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APPENDIX  A 


CURRENT  PERFORMANCE  MEASUREMENTS 

In  1971,  the  FESO  initiated  measurements  to  gauge  the 
activity  of  the  Assistance  Program  and  its  usage  by  field 
personnel.  The  Assistance  Program  is  currently  measured  by 
parameters,  which  are  indicators  of  activity,  rather  than 
effectiveness.  For  the  most  part,  they  measure  level  of 
effort  or  input,  rather  than  output. 

A.  NUMBER  OF  REQUESTS 

The  number  of  requests  measurement  is  simply  a count  of 
the  individual  assistance  requests  received  from  personnel 
located  at  shore  activities.  This  is  one  of  the  two  primary 
measures  used  to  gauge  usage  of  the  Assistance  Program.  A 222 
percent  increase  in  usage  over  the  six  years  that  measurements 
have  been  made  is  encouraging.  The  results  of  these  measure- 
ments are  shown  in  Figure  10.  During  the  fourth  quarter  of 
FY-76,  145  requests  were  received  from  shore  activities. 

B.  NUMBER  OF  ACTIVITIES 

The  number  of  activities  measurement  is  a count  of  the 
individual  activities  whose  personnel  have  requested  assist- 
ance. This  is  the  other  primary  measure  for  evaluating  usage 
of  the  program.  These  two  measure  were  selected  and  are  used 
together,  because  it  is  felt  that  it  is  better,  in  terms  of 
value  to  the  Navy,  to  receive  one  request  from  each  of  ten 
activities,  than  ten  requests  from  one  activity.  The  magnitude 
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Number  of  Assistance  Requests 


of  the  number  of  activities  count  is  smaller  than  the  number 
of  requests,  simply  because  there  are  more  people  than  there 
are  shore  activities.  A 155  percent  increase  in  usage,  over 
the  same  six-year  measuring  period,  is  also  encouraging.  The 
results  of  these  measurements  are  shown  in  Figure  11.  During 
the  fourth  quarter  of  FY-76,  requests  were  received  from 
personnel  at  78  individual  shore  activities. 

C.  SOURCE  OF  REQUESTS 

The  source  of  requests  measurement  categorizes  the  number 
of  requests  received  by  the  type  of  organization  that  the 
individual  requestor  represents.  The  source  of  requests  by 
type  of  organization  for  requests  received  during  the  fourth 
quarter  of  FY-76  are  shown  in  Figure  12. 

D.  METHOD  OF  REQUEST 

The  method  of  receipt  categorizes  the  method  by  which 
the  individual  requests  were  initially  received  by  CEL 
personnel.  Information  on  the  method  of  receipt  for  requests 
received  during  the  fourth  quarter  of  FY-76  is  shown  in 
Figure  13.  The  fact  that  88  percent  of  the  requests  were 
initially  received  by  telephone  is  an  encouraging  acceptance 
of  CEL's  philosophy  that  it  should  be  easily  reached  by 
telephone . 

E.  SUBJECT  OF  REQUESTS 

The  subject  of  requests  measurement  is  a categorization 
of  the  number  of  requests  received  into  broad  subject  cate- 
gories. This  measurement  is  expressed  as  a percentage  and 


70 


71 


i’igure  12  Summary  of  Requests  by  Major  Requestor  - Report  #4 


MEDIUMIN  MEDIUM  IN 


Figure  13  Method  of  Receiving  Requests  - Report  #3 


tends  to  illustrate  that  the  full  breadth  of  CEL's  capabili- 
ties are  employed  in  providing  these  services  to  shore  acti- 
vities. The  subjects  of  requests  received  during  the  fourth 
quarter  FY-76  are  shown  in  Figure  14. 

F.  RESPONSE  TIME 

The  response  time  measurement  is  simply  a count  of  the 
calendar  days  that  elapse  after  receipt  of  a request  by  CEL, 
until  it  is  satisfied.  This  measure  is  the  only  one  that 
approaches  a measurement  of  output — timeliness  of  response. 

The  results  of  this  measurement  for  requests  received  during 
the  fourth  quarter  FY-76  are  shown  in  Figure  15.  This  is  a 
display  of  the  elapsed  time  to  satisfy  a request  by  the  number 
of  requests  satisfied  within  that  period  of  time.  The  fact 
that  72  percent  of  the  requests  received  during  the  fourth 
quarter  were  satisfied  within  seven  calendar  days  speaks  well 
of  the  capability  of  CEL  personnel  to  respond  to  field  needs 
in  a timely  manner. 

These  measurements  are  useful  in  that  they  indicate  an 
active  and  growing  program,  but  with  one  exception  they  suffer 
from  the  disadvantage  of  measuring  effort.  Additional  param- 
eters, that  provide  meaningful  measures  of  output  and  benefit 
are  now  required. 

One  current  shortcoming  in  the  administration  of  the 
Assistance  Program  is  the  lack  of  organized  feedback  from  its 
users.  An  effectiveness  measuring  system  will  provide  a lim- 
ited amount  of  feedback  information,  but  on  a continuing  basis. 
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RE8PQNSE  REUUEST  RE8PUNSE  F IME  BY  IT8ELE 


Figure  15  Response  Time  - Report  #1 


This  information  will  be  useful  in  guiding  and  improving 
the  program  and  expanding  the  potential  for  multiple  use  of 
certain  successful  applications  of  research  results. 
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APPENDIX  B 


ADDITIONAL  BENEFITS  OF  THE  ASSISTANCE  PROGRAM 


The  benefits  of  the  Assistance  Program  are  many  and  varied, 
and  personnel  at  CEL  and  shore  activities  both  share  in  these 
benefits.  Since  this  study  dealt  only  with  those  benefits 
that  could  be  measured  to  some  quantitative  degree,  it  was 
considered  important  to  list  them  so  that  they  will  not  be  over- 
looked and  forgotten.  Some  of  the  additional  benefits  of  the 
Assistance  Program  are  that  it: 

- Promotes  two-way  communication  and  hence  better  under- 
standing between  laboratory  and  field  personnel. 

- Accelerates  the  diffusion  and  adoption  of  new  knowledge. 

- Provides  valuable  contact  for  laboratory  personnel  with 
actual  field  problems.  Often  this  is  "hands-on"  experi- 
ence gained  when  a site  visit  is  required. 

- Identifies  actual  field  needs  that  would  be  amenable  to 
solution  through  the  R&D  process. 

- Provides  feedback  information  to  those  involved  in  the 
R&D  process,  which  serves  to  guide  the  direction  of  the 
research  program  toward  efforts  related  to  common  field 
needs,  the  solutio"  of  which  would  have  far-reaching 
impact.  Although  there  was  no  attempt  made  to  assess 
the  value  of  this  benefit,  there  is  little  doubt  that  it 
exists.  Some  future  effort  might  attempt  to  evaluate 
the  increased  effectiveness  of  a research  program  as  a 
result  of  having  better  knowledge  of  the  actual  needs 

of  the  end  users  of  the  research.  For  instance,  an 
improvement  of  between  one  and  two  percent  of  CEL's 
program  would  offset  the  annual  cost  of  the  Assistance 
Program. 

- Provides  information  not  otherwise  available  to  field 
personnel . 

- Saves  the  time  of  field  personnel  who  can  avoid  searching 
for  information  via  literature  searches  and  technical 
reports . 
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- Avoids  the  selection  of  some  less  than  optimum  alterna- 
tives by  field  personnel. 

- Provides  CEL  researchers  with  the  potential  for  almost 
instant  job  satisfaction  through  participation  in  the 
immediate  solution  of  actual  field  problems. 
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APPENDIX  C 


CRITERIA  FOR  INPUT  TO  AND  PERFORMANCE  OF  THE  EVALUATION  PROCESS 

A.  INPUT  CRITERIA 

Before  proceeding  with  the  development  of  a system  model 
to  relate  individual  evaluations  to  an  effectiveness  deter- 
mination, some  criteria  had  to  be  established  to  guide  the 
evaluation  effort.  Since  the  procedures  attempt  to  measure 
benefit  to  users,  the  users  must  be  involved,  but  not  to  the 
extent  that  they  will  view  CEL,  its  personnel,  or  the  evalua- 
tion system  as  a threat  to  themselves  or  to  their  organization. 
Nor  should  they  be  made  to  feel  that  the  effort  is  a busy, 
self-serving  one  on  the  part  of  CEL. 

For  example,  there  is  little  value  in  asking  a person 
questions  related  to  the  benefit  of  the  response  to  a particular 
request,  when  an  experienced  decision  maker  could  subjectively 
estimate  that  little  if  any  quantifiable  benefit  resulted  from 
the  exchange  of  information  between  CEL  and  field  personnel. 
Again,  this  is  not  to  say  that  the  exchange  did  not  provide 
benefit.  It  is  only  a realization  that  quantification  of  some 
benefits  is  not  yet  possible.  If  it  can  be  judged  with  rea- 
sonable  certainty  that  an  assessment  of  benefit  in  quantitative 
terms  would  be  difficult,  if  not  impossible,  why  ask  the  field 
requestor  for  confirmation?  Such  tactics  could  be  interpreted 
as  a threat  to  the  requestor  if  he  thought  that  each  request 
to  CEL  for  assistance  would  be  followed  by  an  evaluation  of 


80 


I 


the  benefit  of  the  response  he  received.  It  might  be  easier 
to  forego  asking  for  assistance.  In  short,  requests  ranked 
by  the  FESO  as  containing  "zero"  benefit  should  not  be  evalu- 
ated in  the  field. 

The  procedures  should  also  involve  the  laboratory  engineers 
and  scientists  who  provide  the  assistance,  but  not  to  the 
extent  that  they  would  view  the  evaluation  effort  as  unneces- 
sary or  as  requiring  an  inordinate  amount  of  their  time.  The 
responsibility  for  reporting  a significant  part  of  the  total 
assistance  activity,  those  requests  which  are  initially  received 
by  telephone,  rests  with  the  research  personnel.  If  the  evalu- 
ation effort  reached  the  point  where  it  consumed  an  inordinate 
amount  of  time,  in  relation  to  that  which  was  required  to  pro- 
vide the  assistance,  it  could  become  easier  to  not  report  the 
assistance  request  and  thereby  automatically  avoid  any  subse- 
quent evaluation  effort.  The  research  personnel  represent  a 
resource  that  makes  the  Assistance  Program  possible.  They 
should  be  protected  and  relieved,  to  the  maximum  extent  pos- 
sible, of  any  unnecessary  administrative  effort  that  detracts 
from  their  research  tasks. 

B.  PERFORMANCE  CRITERIA 

The  question  of  who  should  perform  the  evaluation  of  bene- 
fit of  individual  requests  is  tied  strongly  to  a concern  for 
the  credibility  of  the  results,  the  effort  and  expense  required 
to  perform  evaluations,  and  the  possibility  of  adverse  effects 
on  the  overall  program.  The  advantages  and  disadvantages  of 
some  of  the  methods  considered  are  discussed  below. 
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Evaluation  by  CET  Researchers 

This  method  would  have  considerable  appeal  to  some, 
at  first,  but  there  are  several  major  disadvantages  that 
preclude  it  from  further  serious  consideration.  Although  it 
would  be  administratively  easy  to  initiate,  by  issuing  an 
instruction  requiring  evaluation  of  each  assistance  request, 
the  possibility  of  creating  adverse  effects  would  be  signifi- 
cant. Another  disadvantage  would  be  a complete  lack  of  uni- 
formity of  the  results.  Still  another  disadvantage  is  the 
normal  lack  of  credibility  attached  to  self-evaluation. 

2 . Evaluation  by  Users 

This  method  could  involve  automatically  sending  an 
evaluation  questionnaire  to  each  person  or  activity  that 
requested  assistance.  The  credibility  of  individual  evalua- 
tions would  probably  be  increased,  but  like  evaluation  by  CEL 
researchers,  the  possibility  of  creating  adverse  effects  would 
be  significant.  Also,  the  results  from  self-administered 
questionnaires  would  lack  uniformity.  In  some  cases  thev 
would  not  even  be  returned  which  would  result  in  the  collection 
of  partial,  and  possibly  useless,  data.  Because  the  qualitv 
of  credibility  should  be  strongest  for  data  provided  by  users, 
their  participation  should  be  provided  for  in  the  final  sys- 
tem, but  with  proper  safeguards  to  ensure  adequate  data 
collection. 

3 . Evaluation  by  Contract 

This  would  more  than  likely  be  the  most  expensive 
method  in  terms  of  dollar  costs,  while  affording  the  least 
amount  of  control  over  the  conduct  of  the  evaluation  effort. 
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Thus  the  risk  of  losing  sight  of  the  purpose  for  the  evalua- 
tion and  generating  negative  attitudes  in  the  providers  and 
users  of  the  Assistance  Program  is  high.  Such  results  might 
be  more  believable  to  "outsiders,"  since  the  possible  influ- 
ence of  self-evaluation  would  be  removed,  however,  the  results 
are  primarily  intended  for  internal  NAVFAC  and  CEL  use.  Uni- 
formity of  results  might  be  an  advantage,  but  this  would  de- 
pend upon  how  the  evaluation  effort  was  staffed  and  conducted. 

4 . Evaluation  by  the  FESO 

Given  that  evaluations  are  to  be  performed,  they  could 
not  be  done  by  the  FESO  without  any  input  from  the  users  and 
CEL  personnel.  If  they  were,  the  results  would  be  entirely 
subjective  and  would  lack  any  degree  of  credibility.  However, 
an  evaluation  effort  coordinated  and  administered  by  the  FESO 
has  the  advantage  of  decreasing  the  burden  on  the  CEL  research 
staff,  providing  a high  degree  of  uniformity  of  data,  and  main- 
taining control  over  the  introduction  of  factors  that  would 
result  in  negative  attitudes  being  generated  amcnq  users . 

The  method  of  having  the  evaluation  efforts  adminis- 
tered and  coordinated  by  the  FESO  was  selected  for  the  effec- 
tiveness measuring  model.  With  adequate  input  from  the  users 
and  providers  of  assistance,  this  method  provides  the  control 
necessary  to  ensure  that  sufficient  information  is  provided 
to  NAVFAC  management,  at  an  affordable  price,  and  with  the 
least  disturbance  to  participants  in  the  program. 
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APPENDIX  D 


DEVELOPMENT  OF  A MODIFIED  EVALUATION  PROCEDURE 

A.  MODIFICATION  OF  QUESTIONNAIRE  MEASURING  TECHNIQUE 

As  indicated  earlier,  the  evaluation  technique  developed 
by  Hendrickson  and  Fisher  represented  a breakthrough.  The 
major  input  of  data  used  as  the  basis  for  the  analysis  of  the 
FY-74  assistance  efforts  was  obtained  through  the  use  of  a 
questionnaire  which  was  revised  from  those  used  in  earlier 
studies  (Figure  16) . A detailed  review  of  the  procedures 
and  resulting  data  revealed  that  some  additional  information 
was  gathered  external  to  the  questionnaire  which  necessitated 
some  initial  hand  calculations  prior  to  machine  processing  of 
the  data.  It  was  considered  desirable  to  modify  the  question- 
naire to  avoid  these  external  efforts,  as  discussed  in  the 
following  section.  The  revised  questionnaire  used  in  this 
study  is  shown  in  Figure  17. 

B.  OPTIMIZING  THE  BASIC  EVALUATION  MODEL 

The  basic  evaluation  model  developed  by  Hendrickson  and 
Fisher  quantifies  in  dollar  terms  the  benefit  of  information 
provided  by  CEL  in  response  to  a specific  request  from  a field 
user.  The  model  is  not  subject  to  analytical  analysis  and 
proof,  but  the  results  therefrom  do  have  intuitive  appeal. 

A critical  review  by  this  author  failed  to  reveal  any  short- 
coming in  the  approach  or  logic,  and  it  was  decided  to  use 
the  basic  model,  with  slight  modifications,  to  estimate  the 
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FESO  PROJECT  EFFECTIVENESS  QUESTIONNAIRE 


1.  Description  of  project: 


Do  you  consider  that  the  advice/assistance  received  was  in  any  way 
beneficial?  (If  CEL  contact  led  to  another  source  of  useful  information, 
check  yes  and  include  that  source  in  answering  questions  4 through  7.) 


_(1)  Yes 


(2)  No 


3.  If  no,  why  not?  (Upon  completion  of  this  question,  skip  questions  4 thrcugn 
7.)  (check  one) 

(1)  no  reply  receivea. 

(2)  lab  could  not  provide  expertise  in  this  area. 

(3)  other.  Please  exolain:  


Describe  the  degree  of  utilization  of  the  advice  and/or  assistance. 

(check  as  many  as  are  appropriate) 

(1)  directly  implemented. 

(2)  direct  implementation  is  planned 

(3)  test  and  evaluation  of  recommended  solution  is  planned  or 

underway. 

(4)  more  extensive  research  on  proposed  solution  is  planned 

or  underway. 

(5)  advice  was  not  applicable  to  particular  problem  but  useful 

on  others. 

(6)  problem  solved  witnout  utilization  of  advice/assistance. 


If  implementation  of  advice/assistance  has  been  accomplished  or  is  planned, 
estimate  the  savings,  (answer  as  many  as  are  appropriate) 


0: 

B: 


6.  Do  you  feel  the  solution  to  the  problem  could  be  applicable  to  other 
activities?  (check  one) 


(1 ) Yes  (2)  No 


7.  Many  improvement  projects  have  intangible  benefits  that  are  independent 
of  measurable  dollar  benefits.  If  CEl/RDT&E  advice  and/or  assistance 
was  utilized,  then  estimate  the  intangible  effects,  (check  as  many  as 
are  appropriate) 

(1)  The  technology  was  used  on  more  than  one  project. 

(2)  The  advice  and/or  assistance  provided  stimulus  for 

subsequent  solutions  to  other  projects. 

(3)  Reduced  the  subsequent  severity  of  the  problem. 

(4)  Increased  safety  factor. 

(5)  Increased  morale. 

(6)  Increased  education/training  of  personnel. 

(7)  Dollar  savings  were  only  benefit  derived. 

(2)  Other  intangible  benefits:  


B:: 


8.  Where  did  you  learn  of  the  availability  of  Civil  Engineering  Lab's 
FESO  Program  for  assistance  on  field  problems? 


Name  of  person  originating  request:  

Activity  Position  Autovon  No.  

Name  of  person  filling  out  questionnaire  (if  different  than  originator): 


Figure  16  FESO  Project  Effectiveness  Questionnaire 

Page  2 of  2 
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CEL  ASSISTANCE  FEEDBACK  QUESTIONNAIRE 


1.  Description  of  request: 


2.  Do  you  consider  the  advice/assistance  given  was  in  any  way  beneficial?  (If 
CEL  contact  led  to  another  source  of  useful  information;  check  Yes  and  con- 
sider that  information  in  answering  questions  4 through  8.) 


(1)  Yes 


(2)  No 


If  not,  why  not?  (Upon  completion  of  this  question,  skip  questions  4 through 
3)  (check  one) 

(1)  no  reply  received. 

(2)  no  advice/assistance  available  from  CEL. 

(3)  advice/assistance  given  was  incomplete/ insufficient. 

(4)  advice/assistance  given  didn't  apply  to  problem/project. 

(5)  didn't  agree  with  advice/assistance  given. 

(6)  advice/assistance  received  coo  late. 

(7)  solucion/recommendacion  too  expensive. 

(8)  other.  Please  explain:  


4.  Describe  Che  use  of  the  advice/assistance  provided.  (check  one) 

(1)  directly  implemented. 

(2)  direct  implementation  planned  as  part  of:  (pick  one) 

(21)  funded  project  in  FY  

(22)  budgeted  project  in  FY 

(23)  planned  project  in  FY 

(3)  implementation  being  considered.  (pick  one) 

(31)  test  and  evaluation  is  underway  or  planned. 

(32)  more  extensive  research  on  proposed  solution  is 

underway  or  planned. 

(4)  general  information  for  future  use 

(5)  other.  Please  explain: 


Figure  17  CEL  Assistance  Feedback  Questionnaire 

Page  1 of  2 
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If  implementation  of  advice/assistance  has  been  accomplished,  is  planned,  or 
is  being  considered,  estimate  the  savings.  (answer  as  many  as  are 
appropriate) 


(1)  savings  in  one  time  construction  or  repair  cost.  $ 

(2)  savings  in  maintenance  cost.  S per (cime  period) 

(3)  savings  in  operating  cost.  $ per  (time  period) 

(4)  If  savings  are  not  estimable  in  dollars,  but  were  intangible  (re- 

duced the  severity  of  the  problem,  increased  efficiency,  increased 
safety,  increased  morale  and/or  living  conditions,  etc.)  what  was 
the  dollar  value  of  the  project,  or  portion  of  the  project,  to 
which  the  advice/assistance  applied?  $ 

(5)  other.  Please  explain: 


6.  Describe  the  contribution  of  the  advice/assiscance  to  the  overall  decision, 
(check  one) 

(1)  confirmed  an  earlier  opinion/conclusion. 

(2)  helped  select  between  previously  known  alternative  approaches/ 

solutions. 

(3)  provided  information  used  in  developing  a better  alternative 

approach/solution  chan  was  previously  planned. 

(4)  provided  a recommendation  which  was  a completely  new  (not 

previously  known  locally)  and  better  alternative. 

7.  Of  the  total  information  used  to  make  the  decision  which  resulted  in  the 
above  savings,  whac  portion/percencage  was  provided  by  CEL?  (check  one) 


.20  30 


40  50 


70  30  90  100 


3.  Do  you  feel  the  approach/solution  to  the  problem/project  couid  be  applicable 
to  other  Shore  Activities? 


(1)  Yes 


(2)  So 


Same  of  person  who  originated  request:. 


Activity: . 


Aucovon  So. 


Same  of  person  filling  in  questionnaire  (if  different  chan  originator) 


Activity: . 


Assistance  number: 


Figure  17  CEL  Assistance  Feedback  Questionnaire 
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effectiveness  of  the  Assistance  Program.  A review  of  the 
basic  approach  is  included  along  with  modifications  by  this 
author  to  provide  continuity. 


The  basic  approach  taken  was  to  identify  distinguishable 
categories  into  which  requests  could  be  grouped.  In  so  group- 
ing, consideration  was  given  to  assuring  that  the  methods  of 
quantifying  benefits  within  each  group  was  workable  within 
the  time  resources  of  both  the  evaluator  and  the  persons  sur- 
veyed. Also,  the  number  of  groups  was  limited  to  a meaning- 
ful number  relative  to  the  size  of  the  survey  group.  Thus, 
if  the  requestor  indicated  that  the  information  which  he  had 
received  was  in  any  way  beneficial,  the  request  would  then  be 
categorized  into  the  group  which  best  suited  the  request  at 
hand.  The  groups  ranged  from  easily  identified  benefits, 
through  more  subjective  quantification  methods  to  a final 
unquantif iable  category. 

C.  CATEGORIZATION  OF  REQUESTS 

Hendrickson  and  Fisher  developed  a three-way  categoriza- 
tion of  each  beneficial  request.  The  essence  of  the  categor- 
ization process,  after  determination  of  whether  or  not  a bene- 
fit existed,  reduced  to  the  testing  of  each  request  with  three 
questions : 

1.  (First  Categorization)  Did  the  information  provided 
take  the  form  of  a specific  recommendation  for  solution  of 
a problem,  or  did  it  merely  provide  information  which  was 
combined  with  information  from  other  sources  to  form  a 
basis  on  which  a decision  for  action  (or  in  some  cases,  no 
action)  could  be  based? 

2.  (Second  Categorization)  What  was  the  probability  that 
the  information  provided  will  be  acted  upon?  That  is,  has 
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the  information  been  implemented,  or  is  implementation 
planned,  or  is  implementation  dependent  on  results  of 
tests,  or  is  the  information  being  evaluated  to  determine 
the  advisability  of  implementation? 

3.  (Third  Categorization)  Is  the  realized  or  expected 
benefit,  if  implemented,  estimable  in  terms  such  as  reduced 
cost  or  eliminated  cost? 

This  three-way  categorization  resulted  in  seventeen  dif- 
ferent groups  or  "benefit  codes"  into  which  a given  beneficial 
request  might  be  categorized.  The  complete  breakdown  of  the 
groups  into  which  a request  could  fall  based  on  this  three- 
way  categorization  are  shown  in  Figure  18. 

In  addition,  standard  methods  of  quantifying  the  present 
value  of  expected  future  benefits  for  each  request  were  derived, 
and  an  ordinal  ranking  of  the  subjectivity  of  the  benefit  codes 
was  achieved. 

The  following  subsections  describe  how  the  various  sub- 
jective factors  were  determined  to  be  appropriate  for  each 
phase  of  the  three-way  categorization  process.  Although 
Hendrickson  and  Fisher  developed  mean  values  and  ranges  of 
accuracy,  only  the  mean  values  are  discussed  and  used  in  this 
study,  since  it  is  intended  as  a basis  for  operational  use. 

1 . First  Categorization,  Action  Recommended,  Versus 
Information  Provided 

Hendrickson  and  Fisher  originally  conceived  that  CEL 
would  be  given  100  percent  credit  for  benefits  which  resulted 
from  a specific  action  recommendation.  Benefits  credited  to 
CEL  for  providing  partial  information  on  a problem  were  to  be 
reduced  by  an  appropriate  factor  representing  the  relative 
value  of  the  information.  This  factor  was  expressed  as  a 
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CATEGORIZATION  OF  FY-74  ASSISTANCE  REQUESTS 


FIRST 

CATEGORIZATION 


SECOND 

CATEGORIZATION 


i — Action  Taken 


Action 

Recommended' 


THIRD 

CATEGORIZATION 

C Estimable 

Not  Estimable 


— Test  Required 


Evaluation 


i — Action  Taken 


[—  Action  Planned- 


Information 

Provided 


— Test  Required 


‘ — Evaluation 


General 

Information' 


No  Beneficial 
Information 


Filed  for  Reference 


Counted  as  a Zero  Benefit 


1 — Not  Counted  (Duplicate,  etc. 


BENEFIT 

CODE 

1 

3 


i—  Estimable 

4 

1 — Not  Estimable 

11 

i— Estimable 

7 

1 — Not  Estimable 

13 

r—  Estimable 

9 

' — Not  Estimable 

15 

i—  Estimable 

2 

' — Not  Estimable 

5 

1 — Estimable 

6 

^ — Not  Estimable 

12 

| — Estimable 

8 

' — Not  Estimable 

14 

1 — Estimable 

10 

' — Not  Estimable 

16 

Figure  18  Categorization  of  FY-74  Assistance  Requests 
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percentage  of  the  total  information  required  to  perform  the 
action  necessary  to  realize  the  benefit  in  question.  However, 
the  approach  of  giving  CEL  100  percent  credit  for  all  "Action 
Recommended"  requests  was  abandoned  when  they  realized  that  on 
many  requests  much  of  the  recommendation  incorporated  proce- 
dures which  were  well  known  at  the  field  level. 

They  correctly  realized  that  to  credit  CEL  with  the 
total  benefit  resulting  from  such  a request  would  be  clearly 
inappropriate.  Thus  an  "information-%-factor"  was  applied  to 
every  request.  The  information-%-factor  was  assigned  to  each 
case  by  them  on  an  individual  basis.  The  total  information 
available  on  the  request  was  reviewed  by  Hendrickson  and 
Fisher,  who  had  first-hand  experience  in  Shore  Facilities  Man- 
agement. The  factor  was  selected  from  the  range  of  .01  to  1.00 
based  on  the  relative  effects  the  CEL-provided  information  had 
on  the  selection  of  the  beneficial  alternative  available. 

Other  considerations  were  the  availability  of  the  information 
from  other  sources  and  the  relative  benefit  of  the  next  best 
alternative  that  would  have  been  selected  had  the  CEL-provided 
information  not  been  available.  They  recognized  that  the 
accuracy  of  subjectively  assigning  a relative  percentage  was 
less  for  requests  where  information  was  provided  then  for 
requests  where  a specific  action  recommendation  was  made,  but 
maintained  this  categorization  for  purposes  of  ranking 
subjectivity . 

Since  ranking  subjectivity  would  serve  no  useful  pur- 
pose in  an  on-going  system,  and  an  information-l-factor  was  to 
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be  applied  to  every  beneficial  request,  there  was  no  need  for 
the  first  categorization  in  the  modified  model.  Therefore,  it 
was  deleted. 

It  was  decided  to  gather  data  on  the  value  of  the 
information  provided  by  CEL  direct  from  the  user.  In  this 
way,  the  subjectivity  of  the  procedure  would  be  decreased. 

This  data  was  obtained  via  questions  6 and  7 on  the  revised 
questionnaire  (Figure  17) . Since  question  7 required  a num- 
eric answer,  these  answers  were  used  in  calculating  individual 
benefit.  It  was  considered  that  at  some  later  date  sufficient 
data  would  be  available  to  determine  the  correlation  between 


the  answers  to  questions  6 and  7 and  thereby  develop  some 
reliable  factors  to  apply  to  the  answers  to  question  6.  This 
would  allow  the  potential  obstacle  of  missing  data  to  be  over- 
come should  some  respondent  recognize  the  similarity  of  the 
two  questions  and  elect  to  answer  only  question,  but  not 
question  7. 

Elimination  of  the  first  categorization  simplified 
the  evaluation  model  considerably  in  that  half  of  the  first 
sixteen  benefit  codes  were  deleted.  This  resulted  in  an 


initial  structure  containing  nine  benefit  codes,  including 
the  "General  Information"  benefit  code. 

2 . Second  Categorization,  Probability  of  Implementation 
Requests  for  which  the  response  had  been  considered 
beneficial  were  classified  by  Hendrickson  and  Fisher  into  one 
of  the  following  categories  which  were  adopted  "as  is"  for  use 


in  this  study  and  are  listed  in  descending  order  according  to 
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the  probability  that  the  information  provided  would  be 
implemented: 

1.  Information  had  been  implemented.  (Action  Taken) 

2.  Implementation  is  planned.  (Action  Planned) 

3.  Testing  is  planned  or  underway;  implementation  is 
intended  if  tests  are  successful.  (Test  Required) 

4.  Information  is  being  evaluated;  implementation  is 
intended  provided  evaluation  indicates  that  expected 
benefits  are  probable;  testing  may  be  necessary.  (Evaluation) 

5.  The  information  provided  was  of  a general  type;  not 
related  to  a specific  project.  (General  Information,  Filed 
for  Reference) 

Based  on  their  experience,  Hendrickson  and  Fisher 
assigned  probabilities  to  each  of  the  middle  three  categories 
(2,  3,  and  4 above)  and  that  factor  was  applied  to  the  benefit 
which  would  be  expected  if  the  project  were  implemented.  The 
first  category  required  no  factor  (or  a factor  of  1.0)  since 
implementation  had  already  been  accomplished.  The  fifth  cate- 
gory was  left  unquantified. 

The  probability  value  for  the  middle  three  implementa- 
tion factors:  Planning  factor,  Testing  factor,  and  Evaluation 
factor,  were  selected  and  applied  to  all  requests  as  appropriate. 
The  mean  values  they  agreed  on  for  the  probability  of  these 
implementation  factors  were  as  follows: 


FACTOR 


PROBABILITY 


of 


Planning 

0.5 

Testing 

0.3 

Evaluation 

0.2 

A review  of  these  mean  probability  values  in  the  light 
the  experience  of  the  author  resulted  in  no  reason  to  use 
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other  values,  except  in  the  case  of  the  Planning  factor.  There 
are  several  identifiable  phases  in  the  planning  cycle  which 
should  reasonably  be  expected  to  affect  the  ultimate  fate  of 
a project  in  terms  of  whether  it  is  realized  or  not.  These 
phases  are: 

1.  Funded  projects  for  which  funds  have  been  earmarked, 
although  the  project  has  not  yet  been  implemented. 

2.  Budgeted  projects  which  are  included  in  the  budget 
requests  sent  forward  to  higher  authority  although  not 
yet  acted  upon. 

3.  Planned  projects  which  are  recognized  as  requirements, 
but  whose  importance  is  not  such  that  they  are  included  in 
current  budget  requests. 

The  probability  of  implementation  is  different  for 
projects  in  each  of  these  phases.  There  appeared  to  be  little 
difference  between  a Funded  project  that  is  implemented  and 
one  that  isn't,  in  terms  of  whether  or  not  a decision  will  be 
made  to  use  a particular  piece  of  information.  The  timing  of 
the  decision  is  the  variable,  rather  than  the  probability  that 
the  project  will  be  funded.  Accordingly,  a planning  factor 
of  1.00  was  used  for  those  projects  that  were  funded,  but  not 
yet  implemented. 

At  the  other  end  of  the  scale,  no  plausible  reason 
could  be  found  for  using  an  implementation  factor  other  than 
0.50  for  Planned  projects. 

For  Budgeted  projects,  there  is  a greater  pi 
that  they  will  be  funded  compared  to  Planned  pro  - -•  »• 

lesser  probability  that  they  will  be  funded  c n:  i 

projects.  Thus  the  probability  of  being  ■ •; 

that  of  Funded  and  Planned  projects. 


‘I 


the  system  indicates  that  the  use  of  another  factor  would  be 
more  appropriate,  a factor  of  0.75,  i.e.,  midway  between  the 
other  factors,  was  arbitrarily  selected  for  use. 


This  breakdown  of  the  Action  Planned  category  into 
three  phases  resulted  in  again  expanding  the  number  of  bene- 
fit codes.  Figure  19  shows  the  complete  breakdown  of  the 
benefit  codes  into  which  a request  could  be  categorized. 

3 . Third  Categorization,  Quality  of  Dollar  Estimation 
a.  Estimable  Requests 

Requests  with  dollar  savings  specifically  identi- 
fied on  the  PESO  Project  Effectiveness  Questionnaire  (Figure 
16,  Question  5)  were  classified  by  Hendrickson  and  Fisher 
as  estimable.  If  the  identified  savings  were  of  the  one-time 
type,  versus  recurring,  the  amount  so  identified  was  used  as 
the  request  benefit.  The  benefit  credited  to  CEL  assistance 
for  such  a request  was  the  request  benefit  reduced  by  the  fac- 
tor for  implementation  probability  and  the  information-%-factor 
as  appropriate.  If  the  estimable  identified  savings  were  of 
the  recurring  type,  the  request  benefit  used  was  the  present 
value  of  the  first  five  years  of  savings.  A present  value 
factor  of  3.935  was  used  for  a steady  cash  flow  throughout  the 
year,  utilizing  a 10  percent  rate  of  return.  A present  value 
factor  of  3.977  was  used  for  this  study  to  conform  with  cur- 
rent NAVFAC  guidelines  [Ref.  14].  Again  the  benefit  credited 
to  CEL  assistance  was  the  request  benefit  reduced  by  appropriate 
factors  as  done  above. 


BENEFIT  CODE  CATEGORIES 

BENEFIT 


ESTIMABLE 

Code 

01 

Action 

Taken 

02 

Action 

Planned,  Funded 

03 

Action 

Planned,  Budgeted 

04 

Action 

Planned,  Planned 

05 

Action 

Considered,  T and  E Required 

06 

Action 

Considered,  More  Research  Required 

BENEFIT  NOT 
ESTIMABLE 

Code 

07  Action  Taken 

08  Action  Planned,  Funded 

09  Action  Planned,  Budgeted 

10  Action  Planned,  Planned 

11  Action  Considered,  T and  E Required 

12  Action  Considered,  More  Research  Required 

13  General  Information 

14  No  Benefit 

15  Duplicate  Request 


Figure  19.  Benefit  Code  Categories 
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b.  Not-Estimable  Requests 


1 


Beneficial  requests  which  did  not  have  specifically 
identified  dollar  savings  generally  fell  into  areas  where  the 
benefit  was  in  the  form  of  improved  operations,  better  morale, 
increased  safety,  etc.  In  the  FY-72  and  FY-73  surveys,  bene- 
fits of  requests  of  this  type  were  left  unquantified.  The 
approach  taken  by  Hendrickson  and  Fisher  was  that  with  the  ex- 
ception of  requests  which  fell  into  the  General  Information 
category,  each  request  had  an  identifiable  benefit  even  though 
it  was  not  readily  quantifiable  in  terms  of  direct  dollar 
savings.  They  felt  that  each  could,  in  some  way,  be  identified 
with  a project,  the  magnitude  of  which  was  normally  relatively 
easily  quantified. 

At  this  point,  Hendrickson  and  Fisher  made  an 
assumption  upon  which  a major  portion  of  their  analysis  of  the 
benefit  of  not-estimable  requests  was  based.  They  assumed 
that,  in  order  to  commit  funds  to  a project,  a decision  maker 
must,  whether  he  realized  it  or  not,  expect  a return  in  future 
benefits  which  is  some  percentage  greater  in  present  value 
than  the  initial  outlay.  They  allowed  that  this  percentage 
might  vary  from  decision  maker  to  decision  maker,  but  further 
assumed  that  a reasonable  value  for  this  would  be  ten  percent. 
This  represented  a minimum  expectation.  Many  projects  whose 
expected  return  exceeded  this  minimum  would  be  undertaken, 
and,  of  course,  the  actual  benefit  of  a project  might  fall 
short  of  the  expectation  and  yield  substantially  less  than 
ten  percent.  Nevertheless,  a mean  value  of  ten  percent  seemed 


f ■ - 1 

II  1 1 

intuitively  appropriate  and  consistent  with  rate  of  return 
guidelines  for  cost  benefit  analysis  procedures.  No  fault 
could  be  found  in  this  logic,  and  even  though  it  appeared  to 
be  somewhat  conservative,  there  was  no  logical  basis  for 
selecting  another  value.  Accordingly  it  was  used  in  this 
study. 


i 
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APPENDIX  E 


MODIFICATION  OF  INITIAL  DATA  GATHERING  INSTRUMENT 

In  1971,  procedures  were  formalized  to  monitor  the  flow 
of  information  between  the  field  users  and  the  laboratory  as 
a means  of  developing  information  to  measure  the  usage  of  the 
service  by  field  units.  A form  was  developed  to  gather  inform- 
ation on  assistance  requests  that  were  received  and  answered 
by  telephone  (Figure  20) . Since  the  bulk  of  the  requests  are 
initially  received  by  telephone,  and  many  are  answered  by 
the  same  method,  a method  was  needed  to  allow  convenient  docu- 
mentation and  follow-up  on  their  progress.  To  avoid  placing 
an  inordinate  and  unnecessary  burden  on  research  and  secretarial 
personnel,  that  would  have  resulted  if  formal  memorandum  re- 
porting had  required,  the  form  was  designed  to  allow  comple- 
tion by  the  researcher  in  his  own  handwriting  and  was  printed 
on  carbonless  paper  so  he  could  retain  a record  for  his  own 
purposes.  This  instrument  provided  a convenient  method  to 
record  and  track  those  requests  which  did  not  result  in  some 
form  of  written  communications  with  the  requestor,  and  satis- 
fied the  information  requirements  of  the  researcher  and  the 
FESO  alike.  For  example,  the  data  on  these  forms  was  used 
as  input  to  produce  the  performance  measures  previously  dis- 
cussed and  shown  in  Appendix  A.  However,  efforts  to  evaluate 
benefit  would  require  obtaining  certain  additional  informa- 
tion. It  was  considered  that  some  of  this  information  could 
best  be  gathered  by  CEL  researchers  during  the  initial  call 
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RECORD  OF  RESPONSE  TO  VERBAL  REQUEST  FOR  RDT&E  ASSISTANCE  

NCEl  (T)  38*0/38  (7-71)  date" 

INSTRUCTIONS  FOR  RECORO  OF  RESPONDING  TO  VERBAL  REQUESTS  FOR 
RDT81E  ASSISTANCE  FORM 

1 . This  form  is  for  recording  verbal  requests  for  RDT&E  Assistance  that  do  not  result  m an  official  letter  oemg  sent  to  the  requestor. 

2.  Fill  in  Ov  hand  using  ballpoint  pen,  or  type  if  you  prefer. 

3.  If  request  ts  satisfied  during  the  initial  conversation  with  the  requestor,  complete  items  1 thru  6.  send  original  to  L03C  and  matte  appropriate 
internal  distribution. 

4.  If  further  action  is  required  after  initial  contact,  complete  items  1 thru  5 and  send  copy  to  L03C.  Whoever  is  assigned  action,  complete  item  6. 
send  original  to  LQ3C  and  make  appropriate  internal  distribution. 

5.  Comments  or  suggestions  to  improve  the  form  should  be  directed  to  L03C. 


1.  Received  by: 

name  code 

2.  How  received:  (check  one)  telephone NCEL  visit site  visit_ 

3.  From: 


4.  Details  of  request: 


extension 


telephone 


activity 


5.  Was  request  satisfied  during  initial  contact?  Yes 

a.  If  no,  who  will  complete? 

b.  If  no,  estimated  manhours  to  complete:  0-3. 


over  20_ 


6.  Nature  of  service  performed  or  information  supplied  (include  dates): 


Copy  to:  _ 


L03C 


Figure  20  Record  of  Response  to  Verbal  Request  for 
RDT&E  Assistance 
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for  assistance.  Accordingly,  the  form  was  revised  and  is 
shown  in  Figure  21.  With  the  exception  of  format  changes  to 
increase  the  amount  of  space  available  for  writing,  only  three 
questions  were  added. 

Question  Number  5 was  added  to  obtain  information  that 
would  relate  the  request  to  a particular  type  of  field  pro- 
ject. This  information  would  be  used  to  calculate  benefits, 
since  construction  or  repair  projects  would  result  in  one- 
time benefits,  while  maintenance  and  operating  improvements 
would  generate  annual  benefits  that  would  have  to  be  discounted 
to  present  values. 

Question  Number  8 was  included  to  obtain  a better  assess- 
ment of  the  approximate  cost  of  responding  to  requests  that 
were  initiated  by  telephone.  Accordingly  the  researcher  was 
asked  to  indicate  the  approximate  number  of  hours  expended  on 
a particular  request.  This  information  was  not  used  in  this 
study,  but  provides  input  to  a data  base  for  future  cost/ 
benefit  studies  at  a minimal  collection  cost. 

Question  Number  9 requests  the  researcher  to  make  a purely 
subjective  estimate  concerning  the  possible  value  of  the 
assistance  provided  to  the  requestor.  Since  the  form  was  not 
revised  until  December  1976,  this  type  of  information  was  not 
available  from  CEL  researchers  for  this  study.  However,  it 
will  be  available  for  future  evaluations  to  guide  the  FESO  in 
excluding  specific  requests  from  further  evaluation  on  the 
basis  that  they  do  not  contain  measurable  benefit. 
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RECORD  OF  RESPONSE  TO  VERBAL  REQUEST  FOR  FIELD  ASSISTANCE  

CEL  IT)  3940/38  (12-76) 

1.  Um  form  to  record  verbal  requests  that  do  not  result  in  a latter  reply  to  tha  requestor. 

2.  Ufa  ballpomt  pan.  or  typewriter. 

3.  It  request  i satisfied  during  tha  initial  con  variation,  complata  items  1 thru  9,  sand  orig.  to  L03C  and  make  appropriata  mtarnai  diitnbution. 

4.  It  furthar  action  it  raquirtd  aftar  initial  contact,  complata  itamt  1 thru  6 and  sand  copy  to  L03C.  Whoavar  it  assigned  action,  complata 
itama  7 thru  9,  land  original  to  L03C  and  mala  appropriata  mtarnai  dittribution. 


N*m* 

Cod*  Extention 

From: 

Name 

Cod*  Telephone 

2.  How  Received 

(check  one) 

a Telephone 

□ CEL  visit 

□ Site  visit 


4.  Details  of  request:. 


5.  Type  of  project:  2 Construction 


6.  Was  request  satisfied  during  initial  contact?  3 Yes 

a.  Who  will  complete? 

Name 

7.  Nature  of  service  provided  (include  dates): 


Repair  c Maintenance  3 Operation 
t?  3 Yes  3 No  {answer  quettioni  oelowl 


3 General  Info. 


b.  Est.  manhours  to  complete:  3 0-8  3 8-20 

(check  onal  3 Over  20 


8.  Manhours  charged  (circle  no.l : 0 1 2 3 4 5 8 7 8 9 10  1 1 12  13  14  15  16  17  18  19  20 

9.  To  help  decide  whether  L03C  should  obtain  cost/benefit  information  from  the  requestor,  please  indicate  your  best 
estimate  of  the  potential  cost  savings. 

3 Less  than  $500  3 $500  • S5.000  3 S5.000  - S25.000  3 Over  $25,000 


Copy  to:  L03C .Other:. 

Division 


Date  Completed . 


Figure  21  Record  of  Response  to  Verbal  Request  for 
Field  Assistance 
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APPENDIX  F 


TYPICAL  SAMPLING  CALCULATIONS 


Mean:  The  sample  group  mean  of  the  value  X is  given  by 


1 *• 
-- 


where  X^  = benefit  of  individual  request 

n = number  of  requests  in  sample  group 


Standard  Deviation:  The  sample  standard  deviation  is 


given  by 


- 


2 2X>2 


n - 1 


where  n - 1 = degrees  of  freedom 


Confidence  Limits:  When  small  random  samples  (n  30)  of 
size  n are  taken  from  a normally  distributed  population,  the 
statistic  "t"  has  the  Student's  distribution  with  n - 1 
degrees  of  freedom.  The  95%  confidence  limits  for  "t"  are 
given  by 


fc95  = X rfc  t 


where  t 


.975 


.975  is  the  confidence  coefficient  for 

which  2.5%  of  the  area  lies  in  each 
tail  of  the  "t"  distribution. 
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APPENDIX  F (Continued) 


Multiple  Group  Sampling  - Group  No.  1:  4 requests 


1,150 

0 

8,380 

0 

E = 9,530 
_ 9,530 

X = = 2,382.5 

4 

Sampling  Trial  No.  1:  3 groups  of  4 requests  each 


Group  No. 

Xi 

X 

(xi)2 

1 

17,500 

15.000 

23,862 

30.000 

2,382.5 

5,676,306 

2 

9,943 

0 

0 

39,770 

21,590.5 

466,149,690 

3 

12,428 

154,455,184 

E = 

36,401 

626,281,180 

_ 36,401 

X = = 12,134 

3 
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APPENDIX  F (Continued) 
Samplxng  Trial  No.  1 (Continued) 
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APPENDIX  F (Continued) 


ASSISTANCE 
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ilST 


22  Report  Specification  - 

23  Report  Specification  - 

24  Report  Specification  - 

25  Report  Specification  - 

26  Report  Specification  - 

27  Report  Specification  - 

28  Report  Specification  - 

29  Report  Specification  - 

30  Report  Specification  - 

31  Report  Specification  - 

32  Report  Specification  - 
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ASSISTANCE  PROGRAM  BENEFIT 
REPORT  SPECIFICATIONS 


REPORT  TITLE:  Original  Estimate  of  Total  Benefit 
REPORT  NUMBER:  12 

PURPOSE:  This  report  provides  an  original  estimate  of  total 

benefit  from  all  requests  received  from  shore  activities 
during  the  reporting  period.  It  is  based  on  the  estimated 
ranges  of  potential  benefit  of  each  request  without  prior 
knowledge  of  field  evaluations.  It  will  be  used  to  roughly 
estimate  total  benefit  before  field  evaluations  are  started. 
INPUT  PARAMETERS  FROM  EACH  REQUEST:  Estimates  of  Benefit. 
SELECTION  CONSTRAINTS: 

1.  Include  all  requests  from  shore  activities. 

2.  Include  all  requests  for  selected  quarter. 

DATA  MANIPULATION  REQUIREMENTS: 

1.  Sum  and  print  the  number  of  requests  in  each  range: 

a.  Zero  dollars 

b.  From  1 to  499  dollars 

c.  From  500  to  4,999  dollars 

d.  From  5,000  to  24,999  dollars 

e.  Over  25,000  dollars 

2.  Calculate  and  print  estimated  benefit  in  each  range. 
Multiply  number  of  requests  in  each  range  by  mid-value  of 
respective  range.  Use  25,000  for  "over  25,000  dollar"  range. 

3.  Sum  the  estimated  benefit  in  each  range. 

Figure  22.  Report  Specifications  - Original  Estimate  of 
Total  Benefit 

Page  1 of  2 
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OUTPUT  PARAMETERS: 

1.  Number  of  requests  in  each  range 

2.  Estimated  benefit  in  each  range 

3.  Total  estimated  benefit 

FREQUENCY:  Quarterly 


Figure  22 
Page  2 of  2 
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ASSISTANCE  PROGRAM  BENEFIT 
REPORT  SPECIFICATIONS 

REPORT  TITLE:  Breakdown  of  Evaluation  Status 
REPORT  NUMBER:  13 

PURPOSE:  This  report  summarizes  the  number  of  requests  received 

from  shore  activities  by  whether  and  by  whom  an  evaluation  has 
been  performed,  by  the  estimated  range  of  potential  benefit. 

It  is  used  to  show  the  status  of  evaluation  efforts  for  the 
reporting  period. 

INPUT  PARAMETERS  FROM  EACH  REQUEST:  Evaluation,  Estimate  of 

Benefit 

SELECTION  CONSTRAINTS: 

1.  Include  all  requests  from  shore  activities. 

2.  Include  all  requests  for  selected  quarter. 

DATA  MANIPULATION  REQUIREMENTS: 

1.  Identify  and  sum  the  number  of  requests  in  each  evalu- 
ation phase. 

2.  Identify  and  sum  the  number  of  requests  in  each  range 
of  estimated  potential  value. 

3.  Provide  a matrix  of  Estimate  of  Benefit  and  Evaluation. 

OUTPUT  PARAMETERS:  Number  of  requests  in  each  of  the  above 

categories . 

FREQUENCY:  Quarterly 


Figure  23.  Report  Specifications  - Breakdown  of  Evaluation 

Status  - Report  #13 
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ASSISTANCE  PROGRAM  BENEFIT 
REPORT  SPECIFICATIONS 

REPORT  TITLE:  Breakdown  of  Field  Evaluation  Data 
REPORT  NUMBER:  14 

PURPOSE:  To  determine  the  number  and  type  of  responses  to  the 
feedback  questionnaire  used  to  evaluate  requests  from  shore 
activities.  This  report  provides  background  information  for 
directing  future  program  efforts  and  effectiveness  measurements. 
INPUT  PARAMETERS  FROM  EACH  REQUEST:  Estimate  of  Benefit,  Any 
Benefit,  No  Benefit,  Benefit  Code,  Use  of  Advice  or  Assistance, 
Type  of  Saving,  Contribution  df  Advice  or  Assistance  to  Decision, 
Percent  of  Information  Provided  by  CEL  Compared  to  Total , Would 
Solution  be  Applicable  to  Other  Activities . 

SELECTION  CONSTRAINTS: 

1.  Include  all  shore  activity  requests 

2.  Include  all  requests  for  selected  quarter 

3.  Include  all  requests  which  have  been  evaluated  in  the 
field. 

DATA  MANIPULATION  REQUIREMENTS: 

1.  Identify  and  sum  responses  for  the  above  parameters. 

2.  Determine  frequencies  for  identified  responses. 

OUTPUT  PARAMETERS:  Number  of  requests  in  each  of  above 

categories . 

FREQUENCY : Quarterly 

Figure  24.  Report  Specification  - Breakdown  of  Field 
Evaluation  Data  - Report  #14 
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ASSISTANCE  PROGRAM  BENEFIT 
REPORT  SPECIFICATIONS 


REPORT  TITLE:  Comparison  of  Contribution  and  Information 
Provided 

REPORT  NUMBER:  15 

PURPOSE:  This  report  summarizes  the  field  evaluation  data 

provided  in  response  to  questions  on  requests  received  from 
shore  activities  related  to  the  contribution  of  CEL  assistance 
and  the  percentage  it  represented  compared  to  the  total 
required  to  make  the  decision.  It  will  be  used  to  determine 
the  correlation  between  the  two  parameters  in  a later  attempt 
to  establish  reliable  factors  for  each  of  the  Contribution 
categories ; 

INPUT  PARAMETERS  FROM  EACH  REQUEST:  Contribution  of  Advice  or 
Assistance  to  Decision,  Percentage  of  Information  Provided  by 
CEL  Compared  to  Total. 

SELECTION  CONSTRAINTS: 

1.  Include  all  requests  from  shore  activities. 

2.  Include  all  requests  for  selected  quarter. 

3.  Include  all  requests  which  have  been  evaluated  in  the 
field. 

DATA  MANIPULATION  REQUIREMENTS: 

1.  Identify  and  sum  the  number  of  requests  in  each 
Contribution  category. 

2.  Identify  and  sum  the  number  of  requests  in  each 
Percentage  category. 

3.  Provide  a matrix  of  Contribution  and  Information 
Provided. 

Figure  25.  Report  Specifications  - Comparison  of  Contribution 
and  Information  Provided  - Report  #15 
Page  1 of  2 
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OUTPUT  PARAMETERS:  Number  of  requests  in  each  of  the  above 

categories . 

FREQUENCY:  Quarterly. 


ASSISTANCE  PROGRAM  BENEFIT 
REPORT  SPECIFICATIONS 

REPORT  TITLE:  Summary  of  Savings  by  Type  of  Savings 
REPORT  NUMBER:  16 

PURPOSE:  This  report  provides  the  savings  for  requests  from 
shore  activities  during  the  reporting  period  which  have  been 
evaluated  in  the  field,  by  the  type  of  saving.  This  informa- 
tion will  be  used  to  indicate  how  savings  from  assistance  pro- 
requests are  distributed  among  the  various  types  of  savings. 
INPUT  PARAMETERS  FROM  EACH  REQUEST:  Type  of  Savings,  One- 
Time  Construction  or  Repair  Cost  Saving,  Annual  Maintenance 
Cost  Saving,  Annual  Operating  Cost  Saving,  Project  Cost. 
SELECTION  CONSTRAINTS: 

1.  Include  all  shore  activity  requests. 

2.  Include  all  requests  for  selected  quarter. 

3.  Include  all  requests  which  have  been  evaluated  in  the 
field. 

DATA  MANIPULATION  REQUIREMENTS: 

1.  Identify  and  sum  the  savings  in  each  type  of  savings. 

2.  Identify  and  sum  project  costs. 

3.  Calculate  the  mean  saving  for  each  type  of  saving. 

4.  Calculate  the  mean  project  cost. 

OUTPUT  PARAMETERS: 

1.  Number  of  requests  in  each  category. 

2.  Total  savings  for  each  type  of  savings. 

3.  Total  project  cost. 

4.  Mean  savings  for  each  type  of  savings. 

5.  Mean  project  cost. 

FREQUENCY:  Quarterly 

Figure  26.  Report  Specifications  - Summary  of  Savings  by 

Type  of  Savings  - Report  #16 
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ASSISTANCE  PROGRAM  BENEFIT 
REPORT  SPECIFICATIONS 


REPORT  TITLE:  Estimated  Minimum  Benefit  of  Evaluated  Requests 
REPORT  NUMBER:  17 

PURPOSE:  This  report  provides  an  estimate  of  minimum  total 

benefit  for  requests  from  shore  activities  during  the  reporting 
period,  which  have  been  evaluated  in  the  field.  It  is  based 
on  the  estimated  range  of  potential  benefit  for  each  evaluated 
request.  It  will  be  used  to  determine  the  accuracy  of  estim- 
ating techniques  and  to  determine  the  mid-level  value  for 
requests  with  benefits  over  $25,000. 

INPUT  PARAMETERS  FROM  EACH  REQUEST:  Estimate  of  Benefit. 
SELECTION  CONSTRAINTS: 

1.  Include  all  requests  from  shore  activities. 

2.  Include  all  requests  for  selected  quarter. 

3.  Include  all  requests  which  have  been  evaluated  in  field. 
DATA  MANIPULATION  REQUIREMENTS : 

1.  Sum  the  number  of  requests  in  each  range  of  estimated 
potential  benefit. 

2.  Calculate  and  print  estimated  benefit  in  each  range. 
Multiply  number  of  requests  in  each  range  by  mid-value 
of  respective  range.  Use  25,000  dollars  for  "over 
25,000  dollar"  range. 

3.  Sum  the  estimated  benefit  in  each  range. 


Figure  27.  Report  Specifications  - Estimated  Minimum  Benefit 
of  Evaluated  Requests  - Report  #17 

Page  1 of  2 
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OUTPUT  PARAMETERS:  Number  of  requests  in  each  range. 


1.  Number  of  requests  in  each  range. 

2.  Estimated  benefit  in  each  range. 

3.  Total  estimated  benefit. 

FREQUENCY:  Quarterly. 


Figure  27 
Page  2 of  2 
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ASSISTANCE  PROGRAM  BENEFIT 
REPORT  SPECIFICATIONS 


REPORT  TITLE:  Summary  of  Evaluated  Benefit  by  Type  of  Savings 
REPORT  NUMBER:  18 

PURPOSE:  This  report  provides  the  benefit  of  requests  from 

shore  activities  during  the  reporting  period,  which  have  been 
evaluated  in  the  field,  by  the  type  of  saving.  It  is  based  on 
the  savings  or  project  cost  provided  by  the  user.  It  will  be 
used  to  indicate  how  assistance  benefit  is  distributed  among 
the  various  types  of  savings. 

INPUT  PARAMETERS  FROM  EACH  REQUEST:  Type  of  Saving. 

SELECTION  CONSTRAINTS: 

1.  Include  all  shore  activity  requests. 

2.  Include  all  requests  for  selected  quarter. 

3.  Include  all  requests  which  have  been  evaluated  in  the 
field. 

DATA  MANIPULATION  REQUIREMENTS: 

1.  Calculate  and  print  benefit  for  each  benefit  code  and 
corresponding  type  of  saving. 

2.  Sum  the  benefit  for  each  type  of  saving. 

3.  Calculate  mean  benefit  for  each  type  of  saving  and 
total  benefit. 

OUTPUT  PARAMETERS: 

1.  Number  of  requests  for  each  type  of  saving. 

2.  Benefit  for  each  type  of  saving. 

3.  Mean  benefit  for  each  type  of  saving. 

Figure  28.  Report  Specification  - Summary  of  Evaluated 
Benefits  by  Type  of  Savings  - Report  #18 

Page  1 of  2 
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4.  Total  Benefit. 

5.  Mean  of  total  benefit. 

FREQUENCY:  Quarterly 


Figure  28 
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ASSISTANCE  PROGRAM  BENEFIT 
REPORT  SPECIFICATIONS 

REPORT  TITLE:  Statistics  for  Evaluated  Benefits 
REPORT  NUMBER:  18A 

PURPOSE:  This  report  provides  statistical  analyses  of  the 

benefits  of  requests  from  shore  activities  during  the  reporting 
period,  which  have  been  evaluated  in  the  field,  by  the  type  of 
saving.  It  will  be  used  to  determine  trends,  if  any,  which 
exist  between  various  reporting  periods  in  the  distribution 
of  benefit  among  the  various  types  of  savings. 

INPUT  PARAMETERS  FROM  EACH  REQUEST:  Type  of  Saving 
SELECTION  CONSTRAINTS: 

1.  Include  all  shore  activity  requests. 

2.  Include  all  requests  for  selected  quarter. 

3.  Include  all  requests  which  have  been  evaluated  in  the 
field. 

DATA  MANIPULATION  REQUIREMENT:  The  SPSS  procedures  for  range 
tests,  which  include  the  student  "t"  distribution,  are  used. 
Although  these  tests  are  not  ideally  suited  for  testing  the 
significance  between  the  sampled  groups  of  benefits  used  in 
this  study,  they  will  at  least  provide  an  initial  indication 
of  the  existence  of  any  trends,  when  compared  to  the  results 
of  future  evaluations . 

OUTPUT  PARAMETERS: 

1.  Analysis  of  variance. 

Figure  29.  Report  Specification  - Statistics  for  Evaluated 

Benefits  - Report  #18A 

Page  1 of  2 
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2.  Significance  of  ranges  and  the  differences  between 
the  mean  benefit  for  each  type  of  saving. 

OUTPUT:  Quarterly 


Figure  29 
Page  2 of  2 


ASSISTANCE  PROGRAM  BENEFIT 
REPORT  SPECIFICATIONS 

REPORT  TITLE:  Summary  of  Benefits  by  Benefit  Code 
REPORT  NUMBER:  19 

PURPOSE:  This  report  provides  the  benefit  of  requests  from 

shore  activities  during  the  reporting  period,  which  have  been 
evaluated  in  the  field,  by  the  type  of  benefit.  It  is  based 
on  the  savings  or  project  cost  provided  by  the  user.  It  will 
be  used  to  indicate  how  benefit  is  distributed  among  the  vari- 
ous types  of  benefit. 

INPUT  PARAMETERS  FROM  EACH  REQUEST:  Benefit  Code. 

SELECTION  CONSTRAINTS: 

1.  Include  all  shore  activity  requests. 

2.  Include  all  requests  for  selected  quarter. 

3.  Include  all  requests  which  have  been  evaluated  in  the 
field. 

DATA  MANIPULATION  REQUIREMENTS: 

1.  Calculate  and  print  benefit  for  each  benefit  code  and 
corresponding  type  of  savings. 

2.  Sum  the  benefit  for  each  type  of  benefit. 

3.  Calculate  mean  benefit  for  each  type  of  benefit  and 
total  benefit. 

OUTPUT  PARAMETERS: 

1.  Number  of  requests  for  each  type  of  benefit. 

2.  Benefit  for  each  type  of  benefit. 

3.  Mean  benefit  for  each  type  of  benefit. 

Figure  30.  Report  Specifications  - Summary  of  Benefits  by 

Benefit  Code  - Report  #19 
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ASSISTANCE  PROGRAM  BENEFIT 
REPORT  SPECIFICATIONS 

REPORT  TITLE:  Estimated  Minimum  Remaining  Benefit  by 
Benefit  Estimate 

REPORT  NUMBER:  20 

PURPOSE:  This  report  provides  an  estimate  of  the  minimum 

benefit  of  requests  from  shore  facilities  during  the  reporting 
period  which  have  not  been  evaluated  in  the  office  or  the  field. 
It  is  based  on  the  estimated  range  of  potential  benefit  for 
each  un-evaluated  request.  It  will  be  used  to  estimate  re- 
maining benefit,  which  when  added  to  evaluated  benefit  will 
produce  a more  accurate  value  of  total  benefit  after  evalua- 
tions have  started. 

INPUT  PARAMETERS  FROM  EACH  REQUEST:  Estimate  of  Benefit. 
SELECTION  CONSTRAINTS: 

1.  Include  all  shore  activity  requests. 

2.  Include  all  requests  which  have  not  been  evaluated  in 
office  or  field. 

3.  Include  all  requests  for  selected  quarter. 

DATA  MANIPULATION  REQUIREMENTS: 

1.  Sum  the  number  of  requests  in  each  range  of  estimated 
benefit. 

2.  Calculate  and  print  estimated  benefit  in  each  range. 
Multiply  number  of  requests  in  each  range  by  mid- 
value of  respective  range.  Use  25,000  for  "over 
25,000  dollar"  range. 

3.  Sum  the  estimated  benefit  in  each  range. 

Figure  31.  Report  Specification  - Estimated  Minimum  Remaining 
Benefit  by  Benefit  Estimate  - Report  #20 

Page  1 of  2 
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OUTPUT  PARAMETERS: 


1.  Number  of  requests  in  each  range. 

2.  Estimated  benefit  in  each  range. 

3.  Total  estimated  benefit. 

FREQUENCY:  Quarterly 


Figure  31 


ASSISTANCE  PROGRAM  BENEFIT 
REPORT  SPECIFICATIONS 

REPORT  TITLE:  Comparison  of  Benefit  Estimate  and  Benefit 
REPORT  NUMBER:  21 

PURPOSE:  This  report  summarizes  the  benefit  of  requests 
received  from  shore  activities  during  the  reporting  period 
and  estimated  benefit  of  those  requests.  It  will  be  used  to 
determine  the  correlation  between  actual  and  estimated  bene- 
fit within  ranges  of  benefit. 

INPUT  PARAMETERS  FROM  EACH  REQUEST:  Estimate  of  Benefit. 
SELECTION  CONSTRAINTS: 

1.  Include  all  requests  from  shore  activities. 

2.  Include  all  requests  for  selected  quarter. 

3.  Include  all  requests  which  have  been  evaluated  in  the 
field. 

DATA  MANIPULATION  REQUIREMENTS: 

1.  Calculate  and  print  benefit  for  each  benefit  code  and 
corresponding  type  of  savings. 

2.  Recode  benefits  into  range  of  estimated  benefit. 

3.  Identify  and  sum  the  number  of  benefits  in  each  range 
of  estimated  benefit. 

4.  Identify  and  sum  the  number  of  requests  in  each  range 
of  estimated  benefit. 

5.  Provide  a matrix  of  Benefit  and  Estimate  of  Benefit. 

OUTPUT  PARAMETERS:  Number  of  requests  in  each  of  the  ranges 

of  estimated  benefit. 

FREQUENCY:  Quarterly 

Figure  32.  Report  Specification  - Comparison  of  Benefit 
Estimate  and  Benefit  - Report  #21 
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APPENDIX  H 

ASSISTANCE  PROGRAM  DATA  DEFINITION  CARDS 


Assistance  Program  Data  Definition  Cards 
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Program  Data  Definition  Cards 
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Figure  39  Assistance  Benefit  Report  #18  - Task  Definition  Logic 
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Figure  42  Assistance  Benefit  Report  #20  - Task  Definition  Logic 
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1'igure  44  Assistance  Program  Benefit,  Original  Estimate  of  Total  Benefits  - Report  #12 
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Figure  45  Assistance  Program  Benefit,  Breakdown  of  Evaluation  Status  - Report  #13 


Figure  46  Assistance  Program  Benefits,  Breakdown  of  Field  Evaluation  Data 
Estimate  of  Benefit  - Report  #14 
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Fiyure  46  Assistance  Program  Benefits,  Breakdown  of  Field  Evaluation  Data 
Benefit  Code  - Report  #14 


Assistance  Program  Benefits,  Breakdown  of  Field  Evaluation  Data 
Use  of  Advice  or  Assistance  - Report  #14 
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Figure  46  Assistance  Program  Benefits,  Breakdown  of  Field  Evaluation  Data 
Contribution  of  Advice  or  Assistance  - Report  #14 
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Figure  46  Assistance  Program  Benefits,  Breakdown  of  Field  Evaluation  Data 
Percentage  of  Information  Provided  By  CEL  - Report  #14 
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Figure  46  Assistance  Program  Benefits,  Breakdown  of  Field  Evaluation  Data, 
Would  Solution  Be  Applicable  To  Other  Activities  - Report  #14 
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igure  48  Assistance  Program  Benefit,  Summary  of  Saving  By  Type  of  Savings, 
Annual  Operating  Cost  Saving  - Report  #16 
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Figure  48  Assistance  Program  Benefit,  Summary  of  Saving  By  Type  of  Savings, 
Project  Cost  - Report  #16 
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Figure  50  Assistance  Program  Benefit,  Summary  of  Evaluated  Benefits  By 
Type  of  Benefit  - Report  #18 
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Figure  53  Assistance  Program  Benefit,  Estimated  Minimum  Remaining  Benefit  By 
Benefit  Estimates  - Report  #20 
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